/ttg/ - Tik Tok General

awemer_202... zip
(3.4 MB, 0x0)

more recent awemer version

does anyone have that script for downloading livestreams?

tt_lives_r... zip
(6.45 KB, 0x0)

 >>/3759/
here
but it got stuck on me once
i have to look into it

@lives recorder anon
it would be cool if the program output the .flv link into the console so I could copy paste them to the thread :)
Also, it is working very well, been running for three days straight with no issues

 >>/3950/
you can already get it in two ways
* with -sm and the .json file will have it but the best quality url is hidden in json encoded "stream_data" tag
* ffmpeg output in .out file also has it at the start of the file

But i'll add it the next time i work on it no problem

 >>/4005/
It's merely a convenience thing but I appreciate it anonie

 >>/4005/
first issue I have received 
File "E:\Downloads\awemer\tt_lives_recorder-main\tt_lives_recorder.py", line 144, in main
    with urllib.request.urlopen(request) as url:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 1352, in do_open
    r = h.getresponse()
        ^^^^^^^^^^^^^^^
  File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1374, in getresponse
    response.begin()
  File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\*\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

 >>/4419/
Looks like TikTok api error. If it keeps executing like normal then there's really nothing to do about it.
It should say somewhere "the script will keep executing"

tt_lives_r... zip
(7.13 KB, 0x0)

 >>/4152/
changelog:
* started versioning. assuming first upload was 1.0.0, the one after that 1.1.0 then this is 1.1.1
* fixed(hopefully) a bug where it would get perma stuck on api call (it might be a linux only thing)
* added an example of -f in -h
* added a python version check and meaningful output if the version is too old
* added a "best effort" cookie validity check. has potential for many false positives but all it does is log a warning
* -rt is now 5 by default (was 0). doesn't make sense to disable it, but you can if you want to for whatever reason
* added .flv url in log line when a new live is detected
* added -t/--trace for very verbose logging
* added -q/--quality flag if anybody wants lower quality streams for whatever reason


Also i need a better name for this. I was thinking of "livemer" or "livempy" but this has little in common with awemer.

I will work on tests now before I add any major changes since regression is a thing.

 >>/5529/
cool, thanks for the updates
you could call it flver (pronounced flavor)

anyone else getting severely rate limited?

 >>/5529/
I'm getting an "Api response is logically empty. Cookie might be expired." message quite a bit, even after getting a fresh cookie. Is my account getting cucked or could these be false positives?

 >>/6835/
If your account has nothing going on in your inbox (messages, follow requests, activities) then it's a false positive. Other than calling some other api there is better way to check for cookie unfortunately.

 >>/6729/
a lot more details needed for a reasonable answer

It asks me for a sec_uid when trying to download a user's videos, where do I get that from?

 >>/8443/
sorry I got it

@awemer anon, if you take suggestions
-add flag for downloading user avatar
-add flag for saving video metadata to file, just dump the api response data to a .info.json file, better compared to the description in file name

-add flag for applying video date as mtime to downloaded video file

getting empty response for t=uv
any relaters?

 >>/10596/
yeah seems like it's broken

 >>/10626/
its over

videos-node zip
(28.77 KB, 0x0)

7176270505... mp4
(2.45 MB, 576x1024 h264)

 >>/10596/
multi detail api fails because of the wrong device_type parameter, and single detail apparently needs x-gordon or x-khronos header, but it seems it doesn’t check its value for the time being

 >>/10726/
any idea about t=uv?

 >>/10738/
uv is working fine for me. try to clean/remame request_aweme_cache folder

 >>/10759/
didn't help
what version/command are you using?

 >>/10768/
this version  >>/2600/
modified it here  >>/10726/

node videos-node.js t=uv d=255 q=2 cookie=xxx l=10 i=6921421848353817605 f=./AwemerDownload

 >>/10775/
got it working thanks
so they just started randomly validating device type?

 >>/10726/
if anyone else is retarded enough to use the e command did it also break for you

 >>/10726/
it's not working daamn
2023 sucks 10 dicks

323924944_... jpg
(207.65 KB, 828x1472)

 >>/11095/
nevermind I did this and uv started working again
  >>/10759/
thanks

awemer anon the e flag is no longer working for me with the new version

was anything changed about it?

How do I download both the h264 and h265 version?

 >>/11122/
works for me

 >>/11604/
could you post your commands?
I can get the liked json output file but when I run the command to scrape with it it just outputs another json

I haven't changed my commands at all, just using the newest awemer version

 >>/11122/
input, with liked.json already having been made from a previous command: node videos-node.js o f e="t=uv n=0 j=sid_tt=xxx i={} g=@{}.txt l f=E:\downloads\liked q=2 z d=200 ui l convert_hevc" l d p=10 i=liked.json u;
output, which instead of being awemer downloading toks does this: output file: E:\Downloads\awemer\videos-node.js_20230107T000307Z_cmds.json

I am using the exact same batch file, just with the new version that fixed the empty response errors.

PLS

Some may be interested, yt-dlp added support for downloading Tiktok lives in its latest update
https://github.com/yt-dlp/yt-dlp/releases/tag/2023.01.06

I think I'm doing something wrong,
when I do:

J:\Archives\TikTok\awemer_2022.12.21>node videos-node.js t=v i=7179458700371840298 d o q=2 l metadata_each verify_files f=C:\folder

I get:
Error: Empty response.
    at IncomingMessage. (J:\Archives\TikTok\awemer_2022.12.21\videos-node.js:896:14)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

This is using awemer_2022.12.21 and with the edited videos-node.js that was provided earlier. help?

 >>/11965/
doesn't t=v require cookie to work?

 >>/11967/
I've never used it with cookies and it's always worked for me. /gif/anon here btw (i really need a new nickname)

 >>/12194/
tiktok has changed something. edited version works only with cookie=xxx

tt_lives_r... zip
(8.71 KB, 0x0)

i know i said tests would be next but pythons mocking is kicking my ass and i've had these changes running for a while without issues so might aswell:

changelog:
* did major refactoring
    script is now seperated into following files:
        * arguments.py - argparse and config file parse logic
        * constants.py - global and not so global constants
        * coordination.py - stream recording state and it's control logic.
        * environment.py - environment validation and setup logic
        * post_process.py -  post processing logic. converting merging and temporary file managing
        * recording.py - flv recording logic
        * utils.py - generic utils that don't fit anywhere else
        * tt_lives_recorder.py stream detection/api invoking main loop logic
    this should make the code easier to navigate if you want to look at it (which you should since i'm a stranger posting unverified code on the internet)
* added support for dual streams, will now record streams that someone you follow has joined
    the end result might be you recording some random dude doing nothing for 5hours with your follower having joined for 5 seconds at the start
    no flags to disable it for now
* added a filename mask for quality
* added a seperate log line for tiktok api errors and also a min wait so you can't spam it too much

also -mf with .mp4 files has been working pretty well for me even for very laggy streams

if anybody is still using this let me know how it's working for you and what options you're using

> 11899
looking at the PR I have a suspicion it will not handle stream drops and will not work for all streams in the first place
if that's not the case let me know

 >>/13329/
will try it later, thanks for your work recorderanon

 >>/13329/
> which you should since i'm a stranger posting unverified code on the internet
I will not be doing this

where is the cookie id again i forgor

awemer is rigged with the bonbi virus, it will secretly download bonbi toks in your pc

anyone got a quick rundown of how awemer works?

 >>/14578/
Here's your awemer guide

Download awemer and node.js
Install node and make sure "add to path" is ticked
Run install_dependencies.bat from awemer zip file

Get tokfu's user ID using a UID script like https://pastebin.com/HAqgGMvj (refresh their profile if it doesn't work at first, or look in F12 console)

Open the folder where videos-node.js is stored, click the directory bar, type "cmd" and hit enter
In the newly opened command window type something like this.
You need to fill in your own cookies, from Tiktok.com when you're logged in. Get the cookie by clicking the little lock next to the URL, or using EditThisCookie

node videos-node.js t=uv i=6584010823366344709 d o q=2 l=30 verify_files j=sid_tt=YOURCOOKIEFROMTIKTOK.COM;sessionid=THESAMECOOKIE f=D:\cutest_tokfu

 >>/14602/
Oh and if you have trouble with install_dependencies you can skip it but you won't be able to use verify_files I think.

 >>/14602/
no, actually i'm asking how the code works so that in the future i can try to fix it myself

 >>/14605/
its a huge blob of js callbacks
very difficult to decipher
but you can try if you want the code is right there

 >>/14607/
is it realistic if i try to rewrite it in python?

 >>/14625/
realistic?
no
The scope is too big and the languages are too different.
You could do some basic version in realistic time-frames but the whole functionality of awemer is too vast imo

 >>/12276/
sorry for the late reply. I usually use t=v but I tried using t=uv because t=v didn't work and gave me an "Empty response." error.

Is there any way of downloading stuff now without using cookies from a logged in account? I've never used cookies from tiktok as I don't have an account and it's worked perfectly fine for months.

 >>/16944/
just create an account

 >>/16951/
did that and its working fine

 >>/16959/
only t=v is working though but i only cared about that

is there a way to get username/userid from a video id if the account is private?

> ignoring: Error: Empty response. | 1044
I'm so fucking tired of this shit

looks like t=v is broken again for me (Empty response.)

 >>/18098/
nothing will happen unless t=uv breaks
thats when archive autists get involved

 >>/18098/
just tested, it works for me

 >>/18885/
that's off I'm still getting the same response even now. I'm using the latest awemer, what version are you using?

 >>/18886/
odd *

 >>/18885/
are you using a cookie?

i cant download tiktoks from this profile 7156516182881832235
works fine for all the other profiles ive tried

heres my script 

node videos-node.js t=v d=255 q i=clipboard f="Videos\awemer" cookie=xxx convert_hevc rename_hevc progress_bars x

does it work for you guys?

 >>/18886/
this edited version  >>/10726/
the command is
node videos-node.js cookie=xxx t=v q=2 i=7181239952611544325 f=./download
 >>/18894/
yes, it needs cookie for now

 >>/18895/
fixed it 
it didnt work because the source video is not in HD 
switching to q=2 worked

how do i get userID?

 >>/18944/
dl tampermonkey/violentmonkey extension
use this userscript

// UserScript
// @name         Print UID to Console
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        https://www.tiktok.com/@* [Embed]
// @icon         data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw
// @grant        none
// /UserScript==

(function() {
    'use strict';

    console.log('HELLO!');
    //console.log(NEXT_DATA.props.pageProps.userInfo.user.id);
    let userId = window['SIGI_STATE'].UserModule.users[window['SIGI_STATE'].UserPage.uniqueId].id;
    console.log('HELLO!: ' + userId);
    let infoNode = document.createElement("p");
    infoNode.innerHTML = "Meta: " + userId;// + " " + NEXT_DATA.props.pageProps.userInfo.stats.videoCount;
    //let descNode = document.getElementsByClassName("share-desc")[0];
    //let descNode = document.getElementsByClassName("tiktok-b1wpe9-H2ShareDesc")[0];
    let likesNode = document.getElementsByTagName("strong")[3];
    likesNode.innerHTML =  "Meta: " + userId;
    //let containerNode = likesNode.parentNode;
    //console.log(containerNode, containerNode.childElementCount);
    //containerNode.insertBefore(infoNode, likesNode);
    //console.log(containerNode, containerNode.childElementCount);
    //console.log('DONE');
})();

New Text... txt
(1.32 KB, 0x0)

 >>/18946/

 >>/18946/
 >>/18947/
i already tried this script and it didnt work
prints some red stuff in the console

 >>/18950/
you have to look for the line that says "HELLO" and the UID
sometimes you have to refresh the page

 >>/18951/
here are the outputs from print uid to console

HELLO! Print UID to Console.user.js:15:13
TypeError: window.SIGI_STATE.UserModule is undefined
    VMwrdt7cwtq2p moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/ Print UID to Console.user.js#9:17
    VMwrdt7cwtq2p moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/ Print UID to Console.user.js#9:30
    VMwrdt7cwtq2p moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/ Print UID to Console.user.js#9:31
    Ut moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
     moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/ Print UID to Console.user.js#9:1
    Se moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
    post moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
    ScriptData moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
    onHandle moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
    je moz-extension://57cc0fd2-d46f-472b-9c1c-d2bfcb7a49b7/sandbox/injected-web.js:1
Print UID to Console.user.js:31:28
XHRGEThttps://www.tiktok.com/api/user/detail/?aid=1988&app_language=en&app_name=tiktok_web&browser_language=en-GB&browser_name=Mozilla&browser_online=true&browser_platform=Win32&browser_version=5.0 (Windows)&channel=tiktok_web&cookie_enabled=true&device_id=7188710928320693765&device_platform=web_pc&focus_state=true&from_page=user&history_len=3&is_fullscreen=false&is_page_visible=true&language=en&os=windows&priority_region=&referer=&region=EN&screen_height=1080&screen_width=1920&secUid=MS4wLjABAAAAs2aCZSPn0WmEbk351UV-pmM2TwrIh_2jCNaAR2KiiRCZuQsoC9wQsyzXG-b9NwWF&tz_name=Europe/Paris&uniqueId=jessepinkmanstinkman&webcast_language=en&msToken=JILdwxr4lJXAFB44o0SXx68eMOoJb8pP5VJjeNzLNVqLxlFrOmIpZfkJNdut6NFeaUYr2uGKCFdm-j7Ndfk4p1EVY8mgzcv_Zjqr6r75CyTIiPnBrWIys9pW36h_jvyW5XC56KhDOloDdw==&X-Bogus=DFSzsIVOy-iAN9Y/SZArz2O8kJMp&_signature=_02B4Z6wo000012MB99AAAIDCf0JJby6NiedjAPNAALsO08
[HTTP/2 200 OK 419ms]

 >>/18955/
uhhhh make a new userscript and copy paste what I posted maybe? I have no idea what could be going wrong there.

 >>/18958/
doesnt work
isnt there some other way of getting the userID? i remember you could type some cryptic shit into the console and get it that way, but i dont have it written anywhere.

i tried the extension on Edge and it worked thanks anyways

Capture PNG
(610.18 KB, 908x719)

 >>/18961/
you can also use this extension
https://mega.nz/folder/mIQizApQ#6zAAmcbzIy9dCIkvoOIN2Q

I didn't realize awempy broke cos I only ever scrape likes, but I've updated it to add the cookie flags
https://pastebin.com/fp17iaMf

 >>/18961/
https://pastebin.com/114ujvZa

 >>/13329/
is this currently working? It seems like every cookie I generate gives "WARNING  Api response is logically empty. Cookie might be expired.".

 >>/19975/
just follow some account that is live and see if it records something

 >>/19980/
I did, I get that error message.

 >>/19986/
Maybe its your cookie/account then.
Works for me.
Can you scrape stories with that cookie?
That's not an error just a warning when the response is suspiciously empty. It can false positive if you're account doesn't have a lot going on.

Getting only empty responses even with cookie

 >>/20404/
tikwm broken
sd should still work

 >>/20416/
It's over

 >>/20428/
should be back

Getting an internal server error.
Any idea what might be causing this? Not doing anything different than the last 20 times it worked.


Error: Internal Server Error
    at ClientRequest. (C:\...\awemer_2022.12.10\videos-node.js:834:14)
    at Object.onceWrapper (node:events:628:26)
    at ClientRequest.emit (node:events:513:28)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:693:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:117:17)
    at TLSSocket.socketOnData (node:_http_client:534:22)
    at TLSSocket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)

 >>/22480/
Seems a lot of people are having cookie issues, so I'll just add that I am using a logged in sessionid cookie.

Exact command;
node videos-node.js t=uv d=255 q=2 cookie=2361f490bbc89f8227722d10cd6481c6 l=5000 i=52025260371369984 f=./output

 >>/18946/
 >>/18947/
 >>/18950/
 >>/18955/
 >>/18961/
fixed version
https://pastebin.com/raw/v9UHuwjb

Is there any neat greasemonkey scripts extensions that will just allow me to download a videos on an individual basis as I'm scrolling through a profiles videos. I tried a few on greasyfork but the best and most consistent I got was https://greasyfork.org/en/scripts/447369-tiktok-download . It works but you have to right click open in new tab which is a pain in the put.

 >>/22618/
There's this one https://chrome.google.com/webstore/detail/tikfast-tiktok-video-down/enbpgklkminalibdialjfpkfejckhpfd

But it also opens into a new tab to select download options. It has a bulk feature but I've found it finicky.

image png
(681.48 KB, 828x577)

 >>/22618/
 >>/22619/
don't you guys want to use this extension?  >>/18986/
or do you not trust?

 >>/22625/
If I'm being totally honest, I use this
https://ttdownloader.com/
For single video downloads, or even just a couple videos like when Lily updates.
It's not really what the other guy was asking for, so I didn't recommend it, but it works for me, I find it efficient, and don't mind the extra clicks

 >>/18986/
nice extension
who made this, could you make github page for this extension?

 >>/18986/

Nice, this is amazing

 >>/22682/

https://github.com/giantcarrot/extension-description

Here ya go my nigger

 >>/22824/
 >>/22682/

Actually my bad the code isn't there. It was when I bookmarked it, sorry.

For the first time in weeks I'm getting the error again, despite not scrapping anything for 90 mins. Feels bad, man.