Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/03_guides/14_ai_agents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ Note that:
- `OpenAILike` points `api_base` at the Apify OpenRouter proxy, so no provider API key is needed. It's the LlamaIndex LLM class for any OpenAI-compatible endpoint.
- `Document` wraps each scraped page, and `VectorStoreIndex.from_documents` chunks and embeds them. The index is the retrieval layer that sets LlamaIndex apart from a plain agent loop.
- Embeddings run locally through `HuggingFaceEmbedding` from the `llama-index-embeddings-huggingface` package, so the proxy needs no embeddings endpoint.
- The embedding model downloads on the first run and the Website Content Crawler renders pages in a browser, so this pipeline is heavier than the other examples.
- `index.as_query_engine(output_cls=Answer)` retrieves the passages most relevant to the question, then the LLM composes a validated `Answer` grounded in them.
- Every response carries the passages it used in `response.source_nodes`, so the Actor pushes their `url` metadata as citations. `Actor.call` runs the Website Content Crawler, and `to_document` maps each dataset item to a `Document`, so indexing more content is a matter of crawling more pages. For details, see [Using Apify Actors as tools](#using-apify-actors-as-tools).

Expand Down
16 changes: 13 additions & 3 deletions docs/03_guides/code/14_crewai.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,15 @@ async def main() -> None:
# to fetch the page as clean Markdown.
researcher = Agent(
role='Documentation researcher',
goal='Read the Crawlee docs and note every crawler it describes.',
backstory='A researcher who reads technical docs closely.',
goal=(
'Fetch the page with the web_browser tool and note every crawler '
'it describes.'
),
backstory=(
'A researcher who never answers from memory. Before writing '
'anything, always read the actual page with the web_browser tool '
'and report only what it says.'
),
tools=[WebBrowserTool()],
llm=llm,
)
Expand All @@ -91,7 +98,10 @@ async def main() -> None:
)

research = Task(
description=f'Scrape {url} and list the crawlers the page covers.',
description=(
f'Use the web_browser tool to fetch {url}. Based only on the returned '
'Markdown, list the crawlers the page covers. Do not use prior knowledge.'
),
expected_output='Notes on each crawler: name, what it builds on, its use.',
agent=researcher,
)
Expand Down
2 changes: 1 addition & 1 deletion website/roa-loader/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ async function getHash(source) {
build: 'latest',
contentType: 'application/json; charset=utf-8',
memory,
timeout: 180,
timeout: 360,
},
}),
headers: {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ Note that:
- `OpenAILike` points `api_base` at the Apify OpenRouter proxy, so no provider API key is needed. It's the LlamaIndex LLM class for any OpenAI-compatible endpoint.
- `Document` wraps each scraped page, and `VectorStoreIndex.from_documents` chunks and embeds them. The index is the retrieval layer that sets LlamaIndex apart from a plain agent loop.
- Embeddings run locally through `HuggingFaceEmbedding` from the `llama-index-embeddings-huggingface` package, so the proxy needs no embeddings endpoint.
- The embedding model downloads on the first run and the Website Content Crawler renders pages in a browser, so this pipeline is heavier than the other examples.
- `index.as_query_engine(output_cls=Answer)` retrieves the passages most relevant to the question, then the LLM composes a validated `Answer` grounded in them.
- Every response carries the passages it used in `response.source_nodes`, so the Actor pushes their `url` metadata as citations. `Actor.call` runs the Website Content Crawler, and `to_document` maps each dataset item to a `Document`, so indexing more content is a matter of crawling more pages. For details, see [Using Apify Actors as tools](#using-apify-actors-as-tools).

Expand Down
16 changes: 13 additions & 3 deletions website/versioned_docs/version-3.4/03_guides/code/14_crewai.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,15 @@ async def main() -> None:
# to fetch the page as clean Markdown.
researcher = Agent(
role='Documentation researcher',
goal='Read the Crawlee docs and note every crawler it describes.',
backstory='A researcher who reads technical docs closely.',
goal=(
'Fetch the page with the web_browser tool and note every crawler '
'it describes.'
),
backstory=(
'A researcher who never answers from memory. Before writing '
'anything, always read the actual page with the web_browser tool '
'and report only what it says.'
),
tools=[WebBrowserTool()],
llm=llm,
)
Expand All @@ -91,7 +98,10 @@ async def main() -> None:
)

research = Task(
description=f'Scrape {url} and list the crawlers the page covers.',
description=(
f'Use the web_browser tool to fetch {url}. Based only on the returned '
'Markdown, list the crawlers the page covers. Do not use prior knowledge.'
),
expected_output='Notes on each crawler: name, what it builds on, its use.',
agent=researcher,
)
Expand Down
Loading