Training Data Sources

Alhena AI supports various types of data sources for bots knowledge-base.

Supported Data Crawling Sources in Alhena AI

Alhena AI supports a wide range of data crawling capabilities, enabling seamless extraction and indexing of data from diverse sources. This article outlines all currently supported crawling integrations.

1. Websites & Web Pages

  • General Websites – HTML pages, blogs, wikis, etc.

  • Sitemap XML – Crawl all URLs listed in an XML sitemap.

  • Confluence Pages – Crawl pages from Atlassian Confluence.

  • Notion Pages – Crawl public Notion pages.

  • Private Notion Pages – With appropriate authentication.

  • ServiceNow Documentation – Crawl content from ServiceNow knowledge base.

2. Google Drive

  • Google Docs

  • Google Sheets (spreadsheets)

  • Google Slides (presentations)

  • Google Drive Folders – Recursive crawling supported.

  • Google Drive Files – Generic support for any file inside Drive.

3. Document Files

  • PDF (.pdf)

  • Word Documents (.doc, .docx)

  • Excel Spreadsheets (.xls, .xlsx)

  • PowerPoint Presentations (.ppt, .pptx)

  • CSV / TSV (.csv, .tsv)

  • Plain Text (.txt), Markdown (.md), RST (.rst)

  • Rich Text Format (.rtf)

  • OpenDocument Files (.odt, .ods, .odp)

  • Apple iWork (.pages, .numbers, .key)

  • Email Files (.eml, .msg)

  • EPUB (.epub), Org-mode (.org)

  • Config/Data Files (.ini, .yaml, .toml, .xml, .json)

  • Images – .jpg, .jpeg, .png, .webp, etc.

4. Video & Media

  • YouTube Videos – Crawled via GeminiVideoScraper or similar.

  • Other Video Files – .mp4, .avi, .mkv, .mov, and more.

5. Social & Communication Platforms

  • Discord Messages

  • Slack Messages

  • Twitter Pages – Accounts, posts.

6. Helpdesk & Ticketing Systems

  • Zendesk Articles

  • Freshdesk Articles

  • Freshservice Articles

  • Gladly Articles

7. Ecommerce Platforms

  • Shopify Products

  • Woocommerce Products

  • Salesforce Commerce Cloud

  • Magento

  • Generic Product Pages – Custom HTML product extraction.

8. GitHub

  • Code Repositories – Source code, documentation, config files.

  • Issues

  • Discussions

9. Custom / Other Sources

  • S3 Uploaded Files

  • Custom Data Sources – Extensible scraper support.

  • PDF Crawling – Via URL or file upload.

  • Document Uploads – Supports any of the formats listed above.

Supported File Extensions (from INCLUDE_PATTERNS)

Source Code

.py, .js, .java, .c, .cpp, .cs, .rb, .go, .rs, .php, .swift, .kt, .ts, .scala, .pl, .r, .sh

Web

.html

Config & Data

.yaml, .yml, .xml, .ini, .toml

Documentation

.md, .rst, .txt

Presentations

.ppt, .pptx, .odp, .key

Spreadsheets

.xls, .xlsx, .ods, .numbers

Archives

.zip, .rar, .7z, .tar, .gz, .bz2, .xz, .iso

Media

.png, .jpg, .jpeg, .gif, .bmp, .tiff, .svg, .webp, .ico, .mp3, .wav, .ogg, .flac, .aac, .m4a, .wma, .mp4, .avi, .mkv, .flv, .mov, .wmv, .m4v, .webm, .vob, .ogv

Other

.pdf, .doc, .docx, .pages, .eml, .msg, .epub, .org, .tsv, .rtf, .dockerfile, etc.

Summary Table

Category
Examples / Platforms Supported

Websites

General, Sitemap, Confluence, Notion, ServiceNow

Google Drive

Docs, Sheets, Slides, Folders, Files

Documents

PDF, Word, Excel, PowerPoint, CSV, TSV, TXT, Markdown, RST, RTF, ODT, ODS, ODP, Pages, Numbers, Key, EML, MSG, EPUB, Org, Images

Video

YouTube, MP4, AVI, MKV, MOV, etc.

Social/Comm

Discord, Slack, Twitter

Helpdesk

Zendesk, Freshdesk, Freshservice, Gladly

Ecommerce

Shopify, Woocommerce, Salesforce, Magento, Generic Product Pages

GitHub

Code, Issues, Discussions

Custom/Other

S3, Custom sources, PDF crawling, Uploads

PreviousTraining StepsNextWebsites

Last updated