OpenAlex - Revision history

Wikisailor: /* OpenAlex Project: Data Recovery & Rebuild Strategy */

2026-03-10T08:37:17Z

OpenAlex Project: Data Recovery & Rebuild Strategy

← Older revision		Revision as of 08:37, 10 March 2026
Line 37:		Line 37:
	\| I want to see 2024 data \|\| /legacy-data/ \|\| Point scripts at /legacy-data/. 2026 files do NOT contain 2024 snapshot records.		\| I want to see 2024 data \|\| /legacy-data/ \|\| Point scripts at /legacy-data/. 2026 files do NOT contain 2024 snapshot records.
	\|-		\|-
	\| I want to use DuckDB on another host \|\| /v2026/parquet_data/ \|\| 1. <code>scp /v2026/parquet_data/* user@host:~/data/</code><br>2. <code>pip install duckdb</code><br>3. Verify with <code>python3 -c "import duckdb; print(duckdb.query(\"SELECT count(*) FROM 'authors.parquet'\").fetchone()[0])"</code>		\| I want to use DuckDB on another host \|\| /v2026/parquet_data/* \|\| 1. <code>scp /v2026/parquet_data/* user@host:~/data/</code><br>2. <code>pip install duckdb</code><br>3. Verify with <code>python3 -c "import duckdb; print(duckdb.query(\"SELECT count(*) FROM 'authors.parquet'\").fetchone()[0])"</code>
	\|-		\|-
	\| I want to move data SAFELY (rsync) \|\| /v2026/parquet_data/ \|\| <code>rsync -avP /mnt/openalex/v2026/parquet_data/ user@remote:~/dest/</code><br>''-P allows resume if connection drops.''		\| I want to move data SAFELY (rsync) \|\| /v2026/parquet_data/ \|\| <code>rsync -avP /mnt/openalex/v2026/parquet_data/ user@remote:~/dest/</code><br>''-P allows resume if connection drops.''

Wikisailor: /* Command Line (Fast Human Fact-Checking) */

2026-03-10T08:34:38Z

Command Line (Fast Human Fact-Checking)

← Older revision		Revision as of 08:34, 10 March 2026
Line 644:		Line 644:
	Simple Join:		Simple Join:
	SELECT a.display_name FROM authors a JOIN works w ON a.id = w.author_id WHERE w.doi = 'X';		SELECT a.display_name FROM authors a JOIN works w ON a.id = w.author_id WHERE w.doi = 'X';
	Verify Version:		Verify Version: (To see how fresh the data is).
	SELECT max(updated_date) FROM works;		SELECT max(updated_date) FROM works;
	~~(To see how fresh the data is).~~

	Top Cited Today:		Top Cited Today:
	SELECT display_name, cited_by_count FROM works WHERE publication_year = 2026 ORDER BY cited_by_count DESC LIMIT 1;		SELECT display_name, cited_by_count FROM works WHERE publication_year = 2026 ORDER BY cited_by_count DESC LIMIT 1;

Wikisailor: /* Command Line (Fast Human Fact-Checking) */

2026-03-10T08:34:00Z

Command Line (Fast Human Fact-Checking)

← Older revision		Revision as of 08:34, 10 March 2026
Line 647:		Line 647:
	SELECT max(updated_date) FROM works;		SELECT max(updated_date) FROM works;
	(To see how fresh the data is).		(To see how fresh the data is).

	Top Cited Today:		Top Cited Today:
	SELECT display_name, cited_by_count FROM works WHERE publication_year = 2026 ORDER BY cited_by_count DESC LIMIT 1;		SELECT display_name, cited_by_count FROM works WHERE publication_year = 2026 ORDER BY cited_by_count DESC LIMIT 1;

Wikisailor: /* Command Line (Fast Human Fact-Checking) */

2026-03-10T08:33:37Z

Command Line (Fast Human Fact-Checking)

← Older revision		Revision as of 08:33, 10 March 2026
Line 645:		Line 645:
	SELECT a.display_name FROM authors a JOIN works w ON a.id = w.author_id WHERE w.doi = 'X';		SELECT a.display_name FROM authors a JOIN works w ON a.id = w.author_id WHERE w.doi = 'X';
	Verify Version:		Verify Version:
	SELECT max(updated_date) FROM works; (To see how fresh the data is).		SELECT max(updated_date) FROM works;
			(To see how fresh the data is).
	Top Cited Today:		Top Cited Today:
	SELECT display_name, cited_by_count FROM works WHERE publication_year = 2026 ORDER BY cited_by_count DESC LIMIT 1;		SELECT display_name, cited_by_count FROM works WHERE publication_year = 2026 ORDER BY cited_by_count DESC LIMIT 1;

Wikisailor: /* Command Line (Fast Human Fact-Checking) */

2026-03-10T08:32:42Z

Command Line (Fast Human Fact-Checking)

← Older revision		Revision as of 08:32, 10 March 2026
Line 631:		Line 631:
	''Best for: Answering a single question in under 5 seconds.''		''Best for: Answering a single question in under 5 seconds.''
	DOI Lookup:		DOI Lookup:
	SELECT ~~title~~ FROM works WHERE doi = 'https://doi.org/10.1038/s41586-020-2012-7';		SELECT display_name FROM works WHERE doi = 'https://doi.org/10.1038/s41586-020-2012-7';
	Count by Year:		Count by Year:
	SELECT count(*) FROM works WHERE publication_year = 2025;		SELECT count(*) FROM works WHERE publication_year = 2025;

Wikisailor: /* Maintenance: Managing the temp_directory */

2026-03-10T08:29:49Z

Maintenance: Managing the temp_directory

← Older revision		Revision as of 08:29, 10 March 2026
Line 356:		Line 356:
	To cleanup, If no scripts are currently running, it is safe to delete these files to reclaim disk space:		To cleanup, If no scripts are currently running, it is safe to delete these files to reclaim disk space:
	rm -rf /mnt/openalex/v2026/duckdb_temp/*.tmp		rm -rf /mnt/openalex/v2026/duckdb_temp/*.tmp
			'''''In high-volume data engineering,''' orphaned temp files are the #1 cause of silent disk failure.''

	===Data Integrity & Verification===		===Data Integrity & Verification===

Wikisailor: /* Maintenance: Managing the temp_directory */

2026-03-10T08:26:14Z

Maintenance: Managing the temp_directory

← Older revision		Revision as of 08:26, 10 March 2026
Line 352:		Line 352:
	* '''⚠️ Critical Maintenance Note:'''		* '''⚠️ Critical Maintenance Note:'''
	** Orphaned Files: If a conversion script crashes, is killed (Ctrl+C), or the VM loses power, DuckDB may leave behind large .tmp files in /mnt/openalex/v2026/duckdb_temp.		** Orphaned Files: If a conversion script crashes, is killed (Ctrl+C), or the VM loses power, DuckDB may leave behind large .tmp files in /mnt/openalex/v2026/duckdb_temp.
	** Action: After every major "Heavy Lift" conversion (Works or Authors), check this directory.		** Action: After every major "Heavy Lift" conversion (Works or Authors), check this directory with the Command:
	** Command:
	ls -lh /mnt/openalex/v2026/duckdb_temp		ls -lh /mnt/openalex/v2026/duckdb_temp
	~~Cleanup:~~		To cleanup, If no scripts are currently running, it is safe to delete these files to reclaim disk space:
	If no scripts are currently running, it is safe to delete these files to reclaim disk space: rm -rf /mnt/openalex/v2026/duckdb_temp/*.tmp		rm -rf /mnt/openalex/v2026/duckdb_temp/*.tmp

	===Data Integrity & Verification===		===Data Integrity & Verification===

Wikisailor: /* The Hierarchy Linker */

2026-03-10T08:24:36Z

The Hierarchy Linker

@@ Line 345: / Line 345: @@
   JOIN '/mnt/openalex/v2026/parquet_data/fields.parquet' f ON sub.field_id = f.id
   JOIN '/mnt/openalex/v2026/parquet_data/domains.parquet' d ON f.domain_id = d.id;</nowiki>
 ===Data Integrity & Verification===

Wikisailor: /* The Hierarchy Linker */

2026-03-10T08:21:53Z

The Hierarchy Linker

← Older revision		Revision as of 08:21, 10 March 2026
Line 335:		Line 335:
	""")		""")
	</nowiki>		</nowiki>
			'''''Wiki Tip:''' Creating the Human-Readable Path''
			* Once extracted, run the following SQL in DBeaver or the CLI to create a master map of the 2026 classification system:
			<nowiki>CREATE VIEW topic_map AS
			SELECT
			sub.display_name as subfield,
			f.display_name as field,
			d.display_name as domain
			FROM '/mnt/openalex/v2026/parquet_data/subfields.parquet' sub
			JOIN '/mnt/openalex/v2026/parquet_data/fields.parquet' f ON sub.field_id = f.id
			JOIN '/mnt/openalex/v2026/parquet_data/domains.parquet' d ON f.domain_id = d.id;</nowiki>

	===Data Integrity & Verification===		===Data Integrity & Verification===

Wikisailor: /* 📊 Monitoring: The Watcher vs. The Web GUI */

2026-03-10T08:16:07Z

📊 Monitoring: The Watcher vs. The Web GUI

← Older revision		Revision as of 08:16, 10 March 2026
Line 161:		Line 161:
	* The Watcher (Terminal): A real-time command-line dashboard.		* The Watcher (Terminal): A real-time command-line dashboard.
	watch -n 60 'curl -s "http://localhost:9200/openalex_works_2026/_count?pretty" \| grep count'		watch -n 60 'curl -s "http://localhost:9200/openalex_works_2026/_count?pretty" \| grep count'
	''Note: If the count stays frozen, the ~~engine~~ is in ~~"High Gear" (buffering)~~. ~~Stop with Ctrl+C~~.''		''Note: OpenSearch uses a Refresh Interval. By default, it only commits data to the searchable index every 1 second, but during heavy ingestion, it may be set to -1 (disabled). If the count stays frozen, the data is in the translog but not yet searchable. Use POST /_refresh to force a count update..''
	* OpenSearch Dashboards (Web GUI): Navigate to http://tayberry:5601. You can use the Dev Tools tab to run queries or the Discover tab to see visual histograms of the papers as they arrive.		* OpenSearch Dashboards (Web GUI): Navigate to http://tayberry:5601. You can use the Dev Tools tab to run queries or the Discover tab to see visual histograms of the papers as they arrive.