Skip to content

Commit a29dc88

Browse files
maxpumperlamaxpumperla
committed
01 examples
Signed-off-by: Max Pumperla <[email protected]>
1 parent 9ad2df0 commit a29dc88

37 files changed

+7057
-0
lines changed

‎_toc.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,30 @@ parts:
6464
- file: courses/00_Developer_Intro_to_Ray/output/05_Intro_Ray_Serve_PyTorch_04.ipynb
6565
- file: courses/00_Developer_Intro_to_Ray/output/05_Intro_Ray_Serve_PyTorch_05.ipynb
6666
- file: courses/00_Developer_Intro_to_Ray/output/README_01.ipynb
67+
- caption: 01 Examples
68+
chapters:
69+
- file: courses/01_examples/output/01_Ray_Data_batch_inference_01.ipynb
70+
- file: courses/01_examples/output/01_Ray_Data_batch_inference_02.ipynb
71+
- file: courses/01_examples/output/01_Ray_Data_batch_inference_03.ipynb
72+
- file: courses/01_examples/output/01_Ray_Data_batch_inference_04.ipynb
73+
- file: courses/01_examples/output/01_Ray_Data_batch_inference_05.ipynb
74+
- file: courses/01_examples/output/01_Ray_Data_batch_inference_06.ipynb
75+
- file: courses/01_examples/output/02_Ray_Data_data_processing_01.ipynb
76+
- file: courses/01_examples/output/02_Ray_Data_data_processing_02.ipynb
77+
- file: courses/01_examples/output/02_Ray_Data_data_processing_03.ipynb
78+
- file: courses/01_examples/output/02_Ray_Data_data_processing_04.ipynb
79+
- file: courses/01_examples/output/02_Ray_Data_data_processing_05.ipynb
80+
- file: courses/01_examples/output/02_Ray_Data_data_processing_06.ipynb
81+
- file: courses/01_examples/output/03_Ray_Serve_online_serving_01.ipynb
82+
- file: courses/01_examples/output/03_Ray_Serve_online_serving_02.ipynb
83+
- file: courses/01_examples/output/03_Ray_Serve_online_serving_03.ipynb
84+
- file: courses/01_examples/output/03_Ray_Serve_online_serving_04.ipynb
85+
- file: courses/01_examples/output/04_Ray_Train_distributed_training_01.ipynb
86+
- file: courses/01_examples/output/04_Ray_Train_distributed_training_02.ipynb
87+
- file: courses/01_examples/output/04_Ray_Train_distributed_training_03.ipynb
88+
- file: courses/01_examples/output/04_Ray_Train_distributed_training_04.ipynb
89+
- file: courses/01_examples/output/04_Ray_Train_distributed_training_05.ipynb
90+
- file: courses/01_examples/output/04_Ray_Train_distributed_training_06.ipynb
6791
- caption: Anyscale 101
6892
chapters:
6993
- file: courses/anyscale_101/output/101_anyscale_intro_jobs_01.ipynb

‎courses/01_examples/01_Ray_Data_batch_inference.ipynb‎

Lines changed: 1141 additions & 0 deletions
Large diffs are not rendered by default.

‎courses/01_examples/02_Ray_Data_data_processing.ipynb‎

Lines changed: 835 additions & 0 deletions
Large diffs are not rendered by default.

‎courses/01_examples/03_Ray_Serve_online_serving.ipynb‎

Lines changed: 368 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,368 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Online Model Serving with Ray Serve\n",
8+
"© 2025, Anyscale. All Rights Reserved"
9+
]
10+
},
11+
{
12+
"cell_type": "markdown",
13+
"metadata": {},
14+
"source": [
15+
"💻 **Launch Locally**: You can run this notebook locally.\n",
16+
"\n",
17+
"🚀 **Launch on Cloud**: Think about running this notebook on a Ray Cluster (Click [here](http://console.anyscale.com/register) to easily start a Ray cluster on Anyscale)"
18+
]
19+
},
20+
{
21+
"cell_type": "markdown",
22+
"metadata": {},
23+
"source": [
24+
"Model serving is the process of deploying machine learning models to production so that they can be accessed and \n",
25+
"used by applications or users. It involves creating an API or interface that allows users to send requests to the model\n",
26+
"and receive predictions in response. There are several libraries and frameworks available for model serving, \n",
27+
"each with its own features and capabilities. In this notebook, we showcase Ray Serve and FastAPI to deploy a sentiment analysis\n",
28+
"machine learning (ML) model."
29+
]
30+
},
31+
{
32+
"cell_type": "markdown",
33+
"metadata": {},
34+
"source": [
35+
"\n",
36+
"### What is Ray Serve?\n",
37+
"Ray Serve is a scalable model serving library that allows you to deploy and manage machine learning models in production.\n",
38+
"With Ray Serve, you can easily create a scalable and distributed serving architecture that\n",
39+
"can handle high traffic and large workloads. It is built on top of Ray, a distributed computing framework\n",
40+
"that allows you to run Python code in parallel across multiple machines. \n",
41+
"Ray Serve provides a simple API for deploying and managing models, as well as features like autoscaling,\n",
42+
"load balancing, and versioning.\n",
43+
"\n",
44+
"Ray Serve is designed to be easy to use and integrate with existing machine learning workflows.\n",
45+
"It supports a wide range of machine learning frameworks, including TensorFlow, PyTorch, and Scikit-learn.\n",
46+
"Ray Serve also provides a simple way to deploy models as REST APIs, using FastAPI,\n",
47+
"making it easy to integrate with web applications and other services.\n",
48+
"\n",
49+
"More information: https://docs.ray.io/en/latest/serve/index.html\n",
50+
"\n",
51+
"\n",
52+
"### Why not use just FastAPI or Flask?\n",
53+
"We could have simply used FastAPI or Flask to create a REST API for the model,\n",
54+
"but Ray Serve provides additional features like autoscaling and load balancing that \n",
55+
"make it a better choice for production deployments. Ray Serve also allows you to easily\n",
56+
"deploy multiple models and manage their versions, which can be useful in a production environment \n",
57+
"where you may need to deploy multiple models or update existing ones.\"\"\""
58+
]
59+
},
60+
{
61+
"cell_type": "markdown",
62+
"metadata": {},
63+
"source": [
64+
"### Outline\n",
65+
"<div class=\"alert alert-block alert-info\">\n",
66+
"<ul>\n",
67+
" <li>Architecture\n",
68+
" <li>Import Libraries\n",
69+
" <li>FastAPI service to accept HTTP requests and scaling with Ray Serve\n",
70+
" <li>Simulate Client: Send test requests\n",
71+
" <li>Shutdown the Serve app and the ray cluster\n",
72+
"</ul>\n",
73+
"</div>"
74+
]
75+
},
76+
{
77+
"cell_type": "markdown",
78+
"metadata": {},
79+
"source": [
80+
"## Architecture"
81+
]
82+
},
83+
{
84+
"cell_type": "markdown",
85+
"metadata": {},
86+
"source": [
87+
"![Architecture Diagram](https://lz-public-demo.s3.us-east-1.amazonaws.com/anyscale101/01_examples/03_Ray_Serve_architecture.svg?sanitize=true)"
88+
]
89+
},
90+
{
91+
"cell_type": "markdown",
92+
"metadata": {},
93+
"source": [
94+
"### Import libraries\n",
95+
"In addition to ray and serve, we also import FastAPI to create webservice and Hugging Face transformers to download ML models."
96+
]
97+
},
98+
{
99+
"cell_type": "code",
100+
"execution_count": 1,
101+
"metadata": {},
102+
"outputs": [],
103+
"source": [
104+
"# Import ray serve and FastAPI libraries\n",
105+
"import ray\n",
106+
"from ray import serve\n",
107+
"from fastapi import FastAPI\n",
108+
"\n",
109+
"# library for pre-trained models\n",
110+
"from transformers import pipeline"
111+
]
112+
},
113+
{
114+
"cell_type": "markdown",
115+
"metadata": {},
116+
"source": [
117+
"## FastAPI webservice and deploy a model\n",
118+
"FastAPI is used to create a webservice 'app' to accept HTTP requests.\n",
119+
"\n",
120+
"MySentimentModel class loads the ML model and defines *predict* function for online inference. @serve.deployment decorator defines the Ray Serve deployment.\n",
121+
"\n",
122+
"*@app.get()* is used to create a GET '/predict' route. Similarly, @app.post() can be used POST requests. See https://docs.ray.io/en/latest/serve/http-guide.html for more details.\n",
123+
"\n",
124+
"In this example, *application_logic()* function is used to define a sample transformation or business logic that can be applied before sending the input to the ML model for inference. See inline comments for further explanation.\n",
125+
"\n",
126+
"### Scaling deployment\n",
127+
"*num_replicas* parameter sets the number of instances of the deployment. FastAPI and RayServe automatically load balances to send requests to each instance. There are more options to set the *accelerator_type* to GPU and even use fractional GPUs. See configuration options here: https://docs.ray.io/en/latest/serve/configure-serve-deployment.html ."
128+
]
129+
},
130+
{
131+
"cell_type": "code",
132+
"execution_count": 2,
133+
"metadata": {},
134+
"outputs": [],
135+
"source": [
136+
"\n",
137+
"# Define a simple FastAPI app\n",
138+
"app = FastAPI()\n",
139+
"\n",
140+
"# Define a Ray Serve deployment\n",
141+
"# This decorator registers the class as a Ray Serve deployment\n",
142+
"@serve.deployment(num_replicas=2) # num_replicas specifies the number of replicas for load balancing\n",
143+
"@serve.ingress(app) # This decorator allows the FastAPI app to be served by Ray Serve\n",
144+
"class MySentimentModel:\n",
145+
" def __init__(self):\n",
146+
" # Load a pre-trained sentiment analysis model\n",
147+
" self.model = pipeline(\"sentiment-analysis\",\n",
148+
" model=\"distilbert-base-uncased-finetuned-sst-2-english\")\n",
149+
"\n",
150+
" # Define any necessary application logic or transformation logic\n",
151+
" def application_logic(self, text):\n",
152+
" \"\"\" Apply any necessary application logic to the input text.\n",
153+
" \"\"\"\n",
154+
" # simple application logic: truncate text if it exceeds a certain length\n",
155+
" if len(text) > 50:\n",
156+
" return text[:50].lower() # Truncate and convert to lowercase\n",
157+
" else:\n",
158+
" return text.lower()\n",
159+
" \n",
160+
" @app.get(\"/predict\") # Define an endpoint for predictions\n",
161+
" def predict(self, text: str):\n",
162+
" \"\"\" Predict sentiment for the given text.\n",
163+
" \"\"\"\n",
164+
" # Define any necessary application logic or transformation logic\n",
165+
" text = self.application_logic(text) # Apply any necessary application logic to the input text\n",
166+
"\n",
167+
" # Use the model to make a prediction\n",
168+
" result = self.model(text)\n",
169+
" return {\"text\": text, \"sentiment\": result}\n",
170+
" \n",
171+
"\n"
172+
]
173+
},
174+
{
175+
"cell_type": "markdown",
176+
"metadata": {},
177+
"source": [
178+
"\n",
179+
" ### Deploy the model"
180+
]
181+
},
182+
{
183+
"cell_type": "code",
184+
"execution_count": 3,
185+
"metadata": {},
186+
"outputs": [
187+
{
188+
"name": "stderr",
189+
"output_type": "stream",
190+
"text": [
191+
"2025-07-11 09:16:58,624\tINFO worker.py:1908 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n",
192+
"\u001b[36m(ProxyActor pid=56603)\u001b[0m INFO 2025-07-11 09:16:59,927 proxy 127.0.0.1 -- Proxy starting on node c2986ec5ee9361515a52f3c506843d5d247ffc6732250203b8a47f21 (HTTP port: 8000).\n",
193+
"\u001b[36m(ProxyActor pid=56603)\u001b[0m INFO 2025-07-11 09:16:59,946 proxy 127.0.0.1 -- Got updated endpoints: {}.\n",
194+
"INFO 2025-07-11 09:16:59,975 serve 18071 -- Started Serve in namespace \"serve\".\n",
195+
"\u001b[36m(ServeController pid=56591)\u001b[0m INFO 2025-07-11 09:17:00,075 controller 56591 -- Deploying new version of Deployment(name='MySentimentModel', app='default') (initial target replicas: 2).\n",
196+
"\u001b[36m(ServeController pid=56591)\u001b[0m INFO 2025-07-11 09:17:00,178 controller 56591 -- Adding 2 replicas to Deployment(name='MySentimentModel', app='default').\n",
197+
"\u001b[36m(ProxyActor pid=56603)\u001b[0m INFO 2025-07-11 09:17:00,076 proxy 127.0.0.1 -- Got updated endpoints: {Deployment(name='MySentimentModel', app='default'): EndpointInfo(route='/', app_is_cross_language=False)}.\n",
198+
"\u001b[36m(ProxyActor pid=56603)\u001b[0m INFO 2025-07-11 09:17:00,080 proxy 127.0.0.1 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x169597d10>.\n",
199+
"\u001b[36m(ServeReplica:default:MySentimentModel pid=56595)\u001b[0m Device set to use mps:0\n",
200+
"INFO 2025-07-11 09:17:05,114 serve 18071 -- Application 'default' is ready at http://127.0.0.1:8000/.\n",
201+
"INFO 2025-07-11 09:17:05,120 serve 18071 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x337673dd0>.\n"
202+
]
203+
},
204+
{
205+
"data": {
206+
"text/plain": [
207+
"DeploymentHandle(deployment='MySentimentModel')"
208+
]
209+
},
210+
"execution_count": 3,
211+
"metadata": {},
212+
"output_type": "execute_result"
213+
},
214+
{
215+
"name": "stderr",
216+
"output_type": "stream",
217+
"text": [
218+
"\u001b[36m(ServeReplica:default:MySentimentModel pid=56599)\u001b[0m INFO 2025-07-11 09:20:39,996 default_MySentimentModel kcd1ofsq 1d920c68-1c59-4e2e-a64c-04f87d31bab7 -- GET /predict 200 128.0ms\n",
219+
"\u001b[36m(ServeReplica:default:MySentimentModel pid=56595)\u001b[0m INFO 2025-07-11 09:21:11,350 default_MySentimentModel t4o4t2pu dda6dadc-f201-421d-9bbc-638d92b9f065 -- GET /predict 200 543.2ms\n",
220+
"\u001b[36m(ServeReplica:default:MySentimentModel pid=56599)\u001b[0m INFO 2025-07-11 09:21:15,969 default_MySentimentModel kcd1ofsq e3611603-54d1-491a-8b05-1f693f413243 -- GET /predict 200 36.5ms\n",
221+
"\u001b[36m(ServeReplica:default:MySentimentModel pid=56599)\u001b[0m INFO 2025-07-11 09:22:16,285 default_MySentimentModel kcd1ofsq 5be0e806-0bd5-4992-b15d-c59fd52e6195 -- GET /predict 200 422.7ms\n",
222+
"\u001b[36m(ServeReplica:default:MySentimentModel pid=56599)\u001b[0m INFO 2025-07-11 09:22:58,953 default_MySentimentModel kcd1ofsq 8890c548-abf6-49b2-b923-b525fcc403da -- GET /predict 200 76.5ms\n",
223+
"\u001b[36m(ServeController pid=56591)\u001b[0m INFO 2025-07-11 09:23:20,248 controller 56591 -- Removing 2 replicas from Deployment(name='MySentimentModel', app='default').\n",
224+
"\u001b[36m(ServeController pid=56591)\u001b[0m INFO 2025-07-11 09:23:22,283 controller 56591 -- Replica(id='t4o4t2pu', deployment='MySentimentModel', app='default') is stopped.\n",
225+
"\u001b[36m(ServeController pid=56591)\u001b[0m INFO 2025-07-11 09:23:22,283 controller 56591 -- Replica(id='kcd1ofsq', deployment='MySentimentModel', app='default') is stopped.\n"
226+
]
227+
}
228+
],
229+
"source": [
230+
"\n",
231+
"serve.run(MySentimentModel.bind()) # Bind the deployment to the Ray Serve runtime\n"
232+
]
233+
},
234+
{
235+
"cell_type": "markdown",
236+
"metadata": {},
237+
"source": [
238+
"## Simulate Client: Send test requests\n",
239+
"\n",
240+
"We use *requests* library to send HTTP requests to the deployed model.\n",
241+
"\n",
242+
"Note: if you encounter any errors with serve not able to start, most likely it is due to previous instance of serve not being shutdown properly. Restart the notebook or see towards the end of notebook to see how to gracefully shutdown ray serve and the ray cluster."
243+
]
244+
},
245+
{
246+
"cell_type": "code",
247+
"execution_count": 4,
248+
"metadata": {},
249+
"outputs": [],
250+
"source": [
251+
"import requests # used to send HTTP requests to the deployed model"
252+
]
253+
},
254+
{
255+
"cell_type": "code",
256+
"execution_count": 7,
257+
"metadata": {},
258+
"outputs": [
259+
{
260+
"name": "stdout",
261+
"output_type": "stream",
262+
"text": [
263+
"{'text': 'i love ray serve!', 'sentiment': [{'label': 'POSITIVE', 'score': 0.9998507499694824}]}\n"
264+
]
265+
}
266+
],
267+
"source": [
268+
"\n",
269+
"# Query the deployed model\n",
270+
"response = requests.get(\"http://localhost:8000/predict\", params={\"text\": \"I love Ray Serve!\"})\n",
271+
"print(response.json()) # Should print the sentiment analysis result\n"
272+
]
273+
},
274+
{
275+
"cell_type": "code",
276+
"execution_count": 8,
277+
"metadata": {},
278+
"outputs": [
279+
{
280+
"name": "stdout",
281+
"output_type": "stream",
282+
"text": [
283+
"{'text': 'mars landscape is a tough place to live, but it ha', 'sentiment': [{'label': 'NEGATIVE', 'score': 0.9590334296226501}]}\n"
284+
]
285+
}
286+
],
287+
"source": [
288+
"text = \"Mars landscape is a tough place to live, but it has its own beauty. Venus is even tougher, with its extreme heat and pressure. \"\n",
289+
"response = requests.get(\"http://localhost:8000/predict\", params={\"text\": text})\n",
290+
"print(response.json()) # prints result, note the text is truncated to 50 characters in the application logic"
291+
]
292+
},
293+
{
294+
"cell_type": "code",
295+
"execution_count": 9,
296+
"metadata": {},
297+
"outputs": [
298+
{
299+
"name": "stdout",
300+
"output_type": "stream",
301+
"text": [
302+
"{'text': 'national parks are a great way to experience natur', 'sentiment': [{'label': 'POSITIVE', 'score': 0.9996353387832642}]}\n"
303+
]
304+
}
305+
],
306+
"source": [
307+
"text = \"National Parks are a great way to experience nature and wildlife.\"\n",
308+
"response = requests.get(\"http://localhost:8000/predict\", params={\"text\": text})\n",
309+
"print(response.json()) # # prints result, note the text is truncated to 50 characters in the application logic"
310+
]
311+
},
312+
{
313+
"cell_type": "markdown",
314+
"metadata": {},
315+
"source": [
316+
"### Shutdown the Ray Serve instances and Ray Cluster"
317+
]
318+
},
319+
{
320+
"cell_type": "code",
321+
"execution_count": 10,
322+
"metadata": {},
323+
"outputs": [],
324+
"source": [
325+
"# stop ray serve\n",
326+
"serve.shutdown() # Shutdown Ray Serve when done, ray cluster will still be running"
327+
]
328+
},
329+
{
330+
"cell_type": "code",
331+
"execution_count": 11,
332+
"metadata": {},
333+
"outputs": [],
334+
"source": [
335+
"ray.shutdown() # Shutdown Ray cluster"
336+
]
337+
},
338+
{
339+
"cell_type": "markdown",
340+
"metadata": {},
341+
"source": [
342+
"### Summary\n",
343+
"In this notebook, we deployed a sentiment analysis model from Hugging Face using Ray Serve and FastAPI. Using *num_replicas* we scaled the number of instances of the model. There are many more options to autoscale to increase the replicas when the traffic is high and downscale to zero when there is no traffic."
344+
]
345+
}
346+
],
347+
"metadata": {
348+
"kernelspec": {
349+
"display_name": "Python 3 (ipykernel)",
350+
"language": "python",
351+
"name": "python3"
352+
},
353+
"language_info": {
354+
"codemirror_mode": {
355+
"name": "ipython",
356+
"version": 3
357+
},
358+
"file_extension": ".py",
359+
"mimetype": "text/x-python",
360+
"name": "python",
361+
"nbconvert_exporter": "python",
362+
"pygments_lexer": "ipython3",
363+
"version": "3.11.6"
364+
}
365+
},
366+
"nbformat": 4,
367+
"nbformat_minor": 4
368+
}

0 commit comments

Comments
 (0)