Documented spent six months analyzing a sample of 10,000 TikTok posts to understand what risks lie in the ways videos are created and shared by those looking for help on the platform. We built a dataset of videos based on parameters we defined to represent what a New Yorker concerned with immigration issues would see on their TikTok account. Through our findings, we uncovered dozens of copycat accounts ripping off legitimate immigration attorneys, some actively scamming immigrants into sending them money, as well as a band of news influencers circulating tropes about immigrants committing crimes to garner popularity.
TikTok’s unique “For You Page” (FYP) algorithm creates an online environment that promotes videos that catch people's attention over videos from trusted sources. Several profiles we encountered simply download and reupload content from more popular accounts in an attempt to game this algorithm, as their videos can go viral even if they have no followers to begin with. These trends on TikTok make it particularly difficult to track down where content comes from when it gets shared to someone’s feed. TikTok has also placed tight restrictions on systematic analysis of the platform’s content feed, meaning analyzing aggregate trends is not easy within the platform alone.
To conduct our investigation, we developed a technical pipeline that allowed us to take a group of posts; feed them into the code; transcribe, translate, and analyze each of them in aggregate; and cluster them based on their content. This process allowed us to not only organize our massive list of videos by creator and subject matter, but also trace where reposted videos originally came from, and decipher what kinds of accounts are posting them and why.
Our technical pipeline utilizes open-source AI tools — such as the Whisper transcription tool and deep-translator package — but allows you to run these models entirely on your own computer, removing any reliance on commercial AI services and their data centers.
In the process, we realized this technology could be useful to anyone trying to analyze content being shared on TikTok, and so we chose to share our work directly with the public. In this guide, we walk through our open-source code, step by step, as a primer for how others may build a collective sample of TikTok content from their own community and uncover who drives narratives on the platform and how videos pass between actors.
STEP 1: Build a Content Sample Through Collective Data Sharing
TikTok provides a service that allows any user to request their complete video viewing history and download it as a machine-readable data table — sometimes thousands and thousands of records long. Pooling the viewing histories of multiple people together can serve as a comprehensive sample of video trends across a community.
When working with other folks’ personal TikTok data, it’s important that you establish clear lines of consent and clarify exactly how you plan to remove personal information from shared data so that no one with access can trace it back to the original participant. Here are some ideas on how to build your own data sample:
- Draft a one-page note that explains your project and how people can share their anonymized user data to help understand their community on TikTok. Have your newsroom or host organization post this across public channels with a direct contact link to contribute.
- Host a “community data-sharing” event where folks can come participate in person. Talk through your project at the event, and help each individual navigate the request form on TikTok to ask for their viewership data. Make sure to establish digital communications with them so they can send you their user data once they receive it.
- Center your project around an existing community you already belong to — your family, your workplace, your place of worship. Show up in gathering spaces that already exist to ask for participation in the project.
Participants can follow along with this guide in the TikTok app to request their data, but make sure each participant selects these two settings to download their view history with the correct structure:
- select the check box for Your Activity to include data on your video watch history
- select JSON as the file format
A Python script, which can be run either in Google Drive using Google Colab or on your desktop, converts raw user data into a clean spreadsheet to use for analysis. This script removes any personal identifying information while delivering a list of video urls with the dates they were watched.
Note that working with multiple participants — or working with multiple downloaded files when someone’s view history is quite long — may require you to run this script several times over and combine all resulting CSVs into one before proceeding.
STEP 2: Extract TikTok Videos and Metadata From the Web
We developed a scraping tool that takes a list of TikTok video urls and downloads them, one by one, using the open-source yt-dlp tool. The scraper also grabs important metadata such as each video’s author, post date and play count, which allows you to examine the most popular creators in your sample.
The only required input for this scraper is a CSV that includes, at minimum, one “link” column containing complete url links to videos on TikTok — our Python script from STEP 1 sets this up for you.
Scraping videos directly from TikTok can be temperamental: Rate limiting, behavioral tracking and network connectivity issues, among other things, can interfere with the scraping process and leave you with missing extraction videos or, in some cases, just missing audio. Our script keeps a record of each video you download so you can run the scraper in batches, reducing volatility, and our next step can help you spot videos with missing audio in case you want to try scraping those again, too.
STEP 3: Transcribe and Translate Videos Across Languages
Once you have a folder with downloaded TikTok video files, you can use our tool to transcribe each video and translate the transcriptions into English.
Our transcription script uses OpenAI’s whisper tool, an open-source automatic speech recognition (ASR) system that detects audio across dozens of languages and transcribes it in that language. Our translation pipeline then uses Google Translate to generate an English transcript where there isn’t one already. We use the deep-translator library to allow you to easily swap in a different translation service if Google Translate doesn’t meet your standards.
This process keeps an archive of each video’s audio file, which can come in handy when trying to detect which TikTok videos may have been scraped with their sound missing. A quick comparison between the files in your extracted videos folder and audio cache folder can reveal missing audio — and you might want to try rerunning your scraper one more time with a list of these incomplete downloads.
STEP 4: Cluster Videos by Content Similarity
Our last script takes your English video transcripts and clusters them together based on similar content — a key step in helping detect reuploaded videos across our sample. This process uses the DBSCAN machine learning algorithm, specifically designed to group data points based on some kind of “spatial density” index and identify data outliers.
We defined two different notions of “similarity” to help the algorithm group transcripts:
- Containment similarity: catching cases where a clipped or shortened re-upload is a literal sub-segment of a longer original video
- Semantic similarity: capturing videos that convey the same meaning even if the wording differs slightly
Defining “semantic similarity” was important here, even for detecting video reposts. Given the complexity of the content extraction pipeline, subtle differences in audio, particularly across non-English languages, could lead to subtle word differences in transcriptions and translations. Finding exact subsets across transcriptions might miss cases where content was copied. Each of these similarity signals has a “weight” number that you can tweak to influence the overall outcome of the clustering process — whether you want semantic or containment logic to matter more when grouping together video transcripts.
This tool leaves you with a list of “clusters” of videos that all contain, according to the algorithm, matching content. In the case of our TikTok attorneys investigation, this list of clusters showed us entire networks of copycat accounts ripping off legal testimonies from real lawyers. In any example, these clusters will illustrate what kinds of content are considered important enough to copy and disseminate across your community of interest.
Take Our Pipeline to the Next Level
We are providing our code to you free and open source and encourage you to experiment with it and tweak it to fit your particular needs. Consider these ideas for further augmentations of our tools:
- Categorize video transcripts through qualitative analysis with topic modeling from the Gensim library
- Detect false or misleading content using OpenFactVerification
- Detect AI-generated content and compare to TikTok’s in-house AI-detection
- Track repeat faces and figures across the video sample using open-source computer vision algorithms from Open CV
Do something cool and impactful with our code? Let us know! Drop us a line at sam.rabiyah@documentedny.com and lam.vo@documentedny.com.