gecomt/datacaged: datacaged v0.2.0

Résumé fourni par la source

datacaged v0.2.0 🚀 What's new Source migrated to HuggingFace (HTTPS) Data is now sourced from HuggingFace via HTTPS — no more FTP port 21 issues behind firewalls or corporate proxies. New functions caged_update() — incremental update: queries the local DuckDB for the maximum competency already stored, then downloads only what is missing from HuggingFace. The easiest way to keep the database up to date. caged_to_parquet() — exports DuckDB tables to Parquet using DuckDB's native COPY TO ... (FORMAT PARQUET). Supports optional column partitioning (e.g. by uf). Much faster than collect() + arrow::write_parquet() for large volumes. caged_hf_files() — lists available competencies from the HuggingFace repository via its REST API. Replaces caged_ftp_files(). Parallel downloads caged_download() and caged_load() now download MOV, FOR and EXC files simultaneously using future/furrr — approximately 3× faster than sequential mode. caged_load(years = 2023, months = 1:12, workers = 3) # default caged_load(years = 2023, months = 1:12, workers = 1) # sequential options(datacaged.workers = 4) # set globally Stream decompression caged_parse() now reads .7z files directly via archive::archive_read() — no temporary disk extraction. Falls back to disk extraction if streaming fails. Bilingual documentation (EN + PT-BR) All function documentation (?caged_load etc.) is now in English Console messages are in English by default, with Brazilian Portuguese translation via the po/ system 6 vignettes: 3 in English + 3 in Portuguese Platform compatibility | Feature | Windows | macOS | Linux | |---|---|---|---| | Download (HTTPS) | ✅ | ✅ | ✅ | | Parallel downloads | ✅ | ✅ | ✅ | | Novo CAGED (2020+, LZMA) | ✅ | ✅ | ✅ | | Legacy CAGED (pre-2020, PPMd) | ⚠️ requires 7-Zip | ⚠️ requires 7-Zip | ⚠️ requires 7-Zip | ⚠️ Breaking changes Source URL changed: data now comes from HuggingFace, not FTP. Folder names differ: NOVO_CAGED/, CAGED/, CAGED_AJUSTES/. caged_ftp_files() is deprecated — use caged_hf_files(). The old function still works but emits a warning. caged_download.R: .ftp_url() → .hf_url(), .ftp_list() → .hf_list_dir() 📦 Installation # From CRAN install.packages("datacaged") # From GitHub remotes::install_github("gecomt/datacaged") ⚡ Quick start library(datacaged) # Check HuggingFace connectivity caged_status() # Full pipeline — download, parse and write to DuckDB caged_load( years = 2023:2024, months = seq_len(12L), db_path = file.path(tempdir(), "caged.duckdb") ) # Incremental update — only new competencies caged_update(db_path = file.path(tempdir(), "caged.duckdb")) # Query con <- caged_connect(file.path(tempdir(), "caged.duckdb")) dplyr::tbl(con, "caged_mov") |> dplyr::filter(uf == 35) |> dplyr::count(competenciamov) |> dplyr::collect() DBI::dbDisconnect(con, shutdown = TRUE) # Export to Parquet caged_to_parquet( file.path(tempdir(), "caged.duckdb"), output_dir = "~/exports" ) 🔗 Links Documentation HuggingFace dataset Changelog

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.