Crawls
Discover trainable URLs on a website before adding them as sources.
Start a crawl
Starts an asynchronous crawl from a seed URL. Returns 202 Accepted with a task_id; poll the results endpoint until status is completed.
Scope: source:create · Rate limit: Training (20/min)
/organizations/{organization}/crawlsBearer API key. Create one in your dashboard (Settings → API Keys); the full value is shown only once. Requires the organization owner on a premium or active AppSumo plan.
In: header
Path Parameters
Your organization's UUID.
uuidA valid seed URL to crawl from.
uriResponse Body
curl -X POST "https://api.assistloop.ai/api/public/v1/organizations/497f6eca-6276-4993-bfeb-53cbbbba6f08/crawls" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com" }'{
"data": {
"task_id": "celery-crawl-1f9c",
"status": "pending"
},
"meta": {
"request_id": "b7f0b9c2-1a2b-4c3d-8e9f-0a1b2c3d4e5f"
}
}{
"message": "Unauthenticated."
}{
"error": {
"code": "insufficient_scope",
"message": "Token is missing required scope: source:create",
"details": []
},
"meta": {
"request_id": "b7f0b9c2-1a2b-4c3d-8e9f-0a1b2c3d4e5f"
}
}{
"error": {
"code": "validation_failed",
"message": "The content field is required.",
"details": [
{
"field": "content",
"message": "The content field is required."
}
]
},
"meta": {
"request_id": "b7f0b9c2-1a2b-4c3d-8e9f-0a1b2c3d4e5f"
}
}{
"error": {
"code": "upstream_failure",
"message": "Could not complete the request upstream.",
"details": []
},
"meta": {
"request_id": "b7f0b9c2-1a2b-4c3d-8e9f-0a1b2c3d4e5f"
}
}Get crawl results
Returns the crawl's status and, once complete, the discovered URLs. Discovered URLs are filtered to embeddable pages and capped at your plan's remaining link allowance.
Scope: source:read · Rate limit: Polling (200/min)
/organizations/{organization}/crawls/{taskId}Bearer API key. Create one in your dashboard (Settings → API Keys); the full value is shown only once. Requires the organization owner on a premium or active AppSumo plan.
In: header
Path Parameters
Your organization's UUID.
uuidThe asynchronous task ID.
Response Body
curl -X GET "https://api.assistloop.ai/api/public/v1/organizations/497f6eca-6276-4993-bfeb-53cbbbba6f08/crawls/string"{
"data": {
"task_id": "celery-crawl-1f9c",
"status": "completed",
"result": {
"urls": [
"https://example.com/",
"https://example.com/pricing"
],
"skipped_urls": [
"https://example.com/login"
],
"stats": {
"total_urls": 2
}
}
},
"meta": {
"request_id": "b7f0b9c2-1a2b-4c3d-8e9f-0a1b2c3d4e5f"
}
}{
"message": "Unauthenticated."
}{
"error": {
"code": "insufficient_scope",
"message": "Token is missing required scope: source:create",
"details": []
},
"meta": {
"request_id": "b7f0b9c2-1a2b-4c3d-8e9f-0a1b2c3d4e5f"
}
}{
"error": {
"code": "upstream_failure",
"message": "Could not complete the request upstream.",
"details": []
},
"meta": {
"request_id": "b7f0b9c2-1a2b-4c3d-8e9f-0a1b2c3d4e5f"
}
}