Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/CONTRIBUTING
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
Please read https://memos.openmem.net/docs/contribution/overview to learn how to contribute to this repository. 🌟
Please read https://memos-docs.openmem.net/contribution/overview to learn how to contribute to this repository. 🌟

请阅读 https://memos.openmem.net/docs/contribution/overview 了解如何为此项目贡献代码。🌟
请阅读 https://memos-docs.openmem.net/contribution/overview 了解如何为此项目贡献代码。🌟
44 changes: 19 additions & 25 deletions .github/ISSUE_TEMPLATE/feature-request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,40 +2,34 @@ name: "\U0001F680 Feature request"
description: Submit a request for a new feature
labels: ["enhancement", "pending"]
body:
- type: markdown
attributes:
value: |
Please do not create issues that are not related to new features under this category.
请勿在此分类下创建和新特性无关的 issues。

- type: checkboxes
id: reminder
id: checklist
attributes:
label: Reminder
description: |
Please ensure you have read the above rules carefully and searched the existing issues.
请确保您已经认真阅读了上述规则并且搜索过现有的 issues。

label: Pre-submission checklist
options:
- label: I have read the above rules and searched the existing issues.
- label: I have searched existing issues and this feature hasn't been requested before | 我已搜索现有问题,确认此功能尚未被请求
required: true
- label: I have read the project documentation and confirmed this feature doesn't already exist | 我已阅读项目文档并确认此功能尚未存在
required: true
- label: This feature request is specific to MemOS and not a general software issue | 该功能请求是针对 MemOS 的,而不是一般软件问题
required: true

- type: textarea
id: description
id: problem
validations:
required: true
attributes:
label: Description
description: |
A clear and concise description of the feature proposal.
请详细描述您希望加入的新功能特性。
label: Problem Statement
placeholder: |
Describe the problem you're trying to solve...
Example: "As a developer using MemOS, I find it difficult to..."

- type: textarea
- type: checkboxes
id: contribution
validations:
required: false
attributes:
label: Pull Request
description: |
Have you already created the relevant PR and submitted the code?
您是否已经创建了相关 PR 并提交了代码?
label: Implementation Contribution
options:
- label: I'm willing to implement this feature myself | 我愿意自己实现此功能
required: false
- label: I would like someone else to implement this | 我希望其他人来实现此功能
required: false
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ evaluation/*tmp/
evaluation/results
evaluation/.env
evaluation/configs/*
**tree_textual_memory_locomo**
.env

# Byte-compiled / optimized / DLL files
Expand Down Expand Up @@ -165,6 +166,7 @@ venv.bak/
*.xlsx
*.json
*.pkl
*.html

# but do not ignore docs/openapi.json
!docs/openapi.json
Expand Down
36 changes: 12 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
<div align="center">
<a href="https://memos.openmem.net/">
<img src="docs/assets/banner_new.gif" alt="MemOS Banner">
<img src="https://statics.memtensor.com.cn/memos/memos-banner.gif" alt="MemOS Banner">
</a>



<h1 align="center">
<img src="docs/assets/memos_logo.png" alt="MemOS Logo" width="50"/> MemOS 1.0: 星河 (Stellar) <img src="https://img.shields.io/badge/status-Preview-blue" alt="Preview Badge"/>
<img src="https://statics.memtensor.com.cn/logo/memos_color_m.png" alt="MemOS Logo" width="50"/> MemOS 1.0: 星河 (Stellar) <img src="https://img.shields.io/badge/status-Preview-blue" alt="Preview Badge"/>
</h1>

<p>
Expand All @@ -19,7 +17,7 @@
<a href="https://pypi.org/project/MemoryOS">
<img src="https://img.shields.io/pypi/pyversions/MemoryOS.svg" alt="Supported Python versions">
</a>
<a href="https://memos.openmem.net/docs/home">
<a href="https://memos-docs.openmem.net/home/overview/">
<img src="https://img.shields.io/badge/Documentation-view-blue.svg" alt="Documentation">
</a>
<a href="https://arxiv.org/abs/2507.03724">
Expand All @@ -31,7 +29,7 @@
<a href="https://discord.gg/Txbx3gebZR">
<img src="https://img.shields.io/badge/Discord-join%20chat-7289DA.svg?logo=discord" alt="Discord">
</a>
<a href="docs/assets/qr_code.png">
<a href="https://statics.memtensor.com.cn/memos/qr-code.png">
<img src="https://img.shields.io/badge/WeChat-Group-07C160.svg?logo=wechat" alt="WeChat Group">
</a>
<a href="https://opensource.org/license/apache-2-0/">
Expand All @@ -42,17 +40,14 @@

---

<a href="https://memos.openmem.net/">
<img src="docs/assets/sota_score.jpg" alt="SOTA SCORE">
</a>

<img src="https://statics.memtensor.com.cn/memos/sota_score.jpg" alt="SOTA SCORE">

**MemOS** is an operating system for Large Language Models (LLMs) that enhances them with long-term memory capabilities. It allows LLMs to store, retrieve, and manage information, enabling more context-aware, consistent, and personalized interactions.

- **Website**: <a href="https://memos.openmem.net/" target="_blank">https://memos.openmem.net/</a>
- **Documentation**: <a href="https://memos.openmem.net/docs/home" target="_blank">https://memos.openmem.net/docs/home</a>
- **API Reference**: <a href="https://memos.openmem.net/docs/api/info" target="_blank">https://memos.openmem.net/docs/api/info</a>
- **Source Code**: <a href="https://github.com/MemTensor/MemOS" target="_blank">https://github.com/MemTensor/MemOS</a>
- **Website**: https://memos.openmem.net/
- **Documentation**: https://memos-docs.openmem.net/home/overview/
- **API Reference**: https://memos-docs.openmem.net/docs/api/info/
- **Source Code**: https://github.com/MemTensor/MemOS

## 📈 Performance Benchmark

Expand All @@ -66,19 +61,12 @@ MemOS demonstrates significant improvements over baseline memory solutions in mu

> 💡 **Temporal reasoning accuracy improved by 159% compared to the OpenAI baseline.**



### Details of End-to-End Evaluation on LOCOMO

> [!NOTE]
> Comparison of LLM Judge Scores across five major tasks in the LOCOMO benchmark. Each bar shows the mean evaluation score judged by LLMs for a given method-task pair, with standard deviation as error bars. MemOS-0630 consistently outperforms baseline methods (LangMem, Zep, OpenAI, Mem0) across all task types, especially in multi-hop and temporal reasoning scenarios.

<a href="https://memos.openmem.net/">
<img src="docs/assets/score_all_end2end.jpg" alt="END2END SCORE">
</a>



<img src="https://statics.memtensor.com.cn/memos/score_all_end2end.jpg" alt="END2END SCORE">

## ✨ Key Features

Expand Down Expand Up @@ -196,7 +184,7 @@ Join our community to ask questions, share your projects, and connect with other
- **Discord**: Join our <a href="https://discord.gg/Txbx3gebZR" target="_blank">Discord Server</a>.
- **WeChat**: Scan the QR code to join our WeChat group.

<img src="docs/assets/qr_code.png" alt="QR Code" width="600">
<img src="https://statics.memtensor.com.cn/memos/qr-code.png" alt="QR Code" width="600">

## 📜 Citation

Expand Down Expand Up @@ -239,7 +227,7 @@ url = {https://global-sci.com/article/91443/memory3-language-modeling-with-expli

## 🙌 Contributing

We welcome contributions from the community! Please read our [contribution guidelines](https://memos.openmem.net/docs/contribution/overview) to get started.
We welcome contributions from the community! Please read our [contribution guidelines](https://memos-docs.openmem.net/contribution/overview) to get started.

## 📄 License

Expand Down
Binary file removed docs/assets/banner_new.gif
Binary file not shown.
Binary file removed docs/assets/memos_logo.png
Binary file not shown.
Binary file removed docs/assets/qr_code.png
Binary file not shown.
Binary file removed docs/assets/score_all_end2end.jpg
Binary file not shown.
Binary file removed docs/assets/sota_score.jpg
Binary file not shown.
4 changes: 3 additions & 1 deletion evaluation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,12 @@ This repository provides tools and scripts for evaluating the LoCoMo dataset usi
## Evaluation Scripts

### LoCoMo Evaluation
To evaluate the **LoCoMo** dataset using one of the supported memory frameworks — `memos`, `mem0`, or `zep` — run the following command:
⚙️ To evaluate the **LoCoMo** dataset using one of the supported memory frameworks — `memos`, `mem0`, or `zep` — run the following [script](./scripts/run_locomo_eval.sh):

```bash
# Edit the configuration in ./scripts/run_locomo_eval.sh
# Specify the model and memory backend you want to use (e.g., mem0, zep, etc.)
./scripts/run_locomo_eval.sh
```

✍️ For evaluating OpenAI's native memory feature with the LoCoMo dataset, please refer to the detailed guide: [OpenAI Memory on LoCoMo - Evaluation Guide](./scripts/locomo/openai_memory_locomo_eval_guide.md).
173 changes: 173 additions & 0 deletions evaluation/scripts/locomo/locomo_openai.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
import argparse
import json
import os
import time

from collections import defaultdict
from multiprocessing.dummy import Pool

from dotenv import load_dotenv
from openai import OpenAI
from tenacity import retry, stop_after_attempt, wait_random_exponential
from tqdm import tqdm


load_dotenv()

# Retry policy constants
WAIT_MIN = 5 # minimum backoff delay in seconds
WAIT_MAX = 30 # maximum backoff delay in seconds
MAX_TRIES = 10 # maximum number of retry attempts

WORKERS = 5 # number of parallel worker processes

ANSWER_PROMPT = """
You are an intelligent memory assistant tasked with retrieving accurate information from conversation memories.

# CONTEXT:
You have access to memories from a conversation. These memories contain
timestamped information that may be relevant to answering the question.

# INSTRUCTIONS:
1. Carefully analyze all provided memories
2. Pay special attention to the timestamps to determine the answer
3. If the question asks about a specific event or fact, look for direct evidence in the memories
4. If the memories contain contradictory information, prioritize the most recent memory
5. If there is a question about time references (like "last year", "two months ago", etc.),
calculate the actual date based on the memory timestamp. For example, if a memory from
4 May 2022 mentions "went to India last year," then the trip occurred in 2021.
6. Always convert relative time references to specific dates, months, or years. For example,
convert "last year" to "2022" or "two months ago" to "March 2023" based on the memory
timestamp. Ignore the reference while answering the question.
7. Focus only on the content of the memories. Do not confuse character
names mentioned in memories with the actual users who created those memories.
8. The answer should be less than 5-6 words.

# APPROACH (Think step by step):
1. First, examine all memories that contain information related to the question
2. Examine the timestamps and content of these memories carefully
3. Look for explicit mentions of dates, times, locations, or events that answer the question
4. If the answer requires calculation (e.g., converting relative time references), show your work
5. Formulate a precise, concise answer based solely on the evidence in the memories
6. Double-check that your answer directly addresses the question asked
7. Ensure your final answer is specific and avoids vague time references

Memories:

{context}

Question: {question}
Answer:
"""


class OpenAIPredict:
def __init__(self, model="gpt-4o-mini"):
self.model = model
self.openai_client = OpenAI(
api_key=os.getenv("OPENAI_API_KEY"), base_url=os.getenv("OPENAI_BASE_URL")
)
self.results = defaultdict(list)

def search_memory(self, idx):
with open(f"openai_memory/{idx}.txt", encoding="utf-8") as file:
memories = file.read().strip().replace("\n\n", "\n")

return memories, 0

def process_question(self, val, idx):
question = val.get("question", "")
answer = val.get("answer", "")
category = val.get("category", -1)

response, search_memory_time, response_time, context = self.answer_question(idx, question)

result = {
"question": question,
"answer": response,
"category": category,
"golden_answer": answer,
"search_context": context,
"response_duration_ms": response_time,
"search_duration_ms": search_memory_time,
}

return result

@retry(
wait=wait_random_exponential(min=WAIT_MIN, max=WAIT_MAX),
stop=stop_after_attempt(MAX_TRIES),
reraise=True,
)
def answer_question(self, idx, question):
memories, search_memory_time = self.search_memory(idx)

answer_prompt = ANSWER_PROMPT.format(context=memories, question=question)

t1 = time.time()
response = self.openai_client.chat.completions.create(
model=self.model,
messages=[{"role": "system", "content": answer_prompt}],
temperature=0.0,
)
t2 = time.time()
response_time = (t2 - t1) * 1000
return response.choices[0].message.content, search_memory_time, response_time, memories

def process_data_file(self, file_path, output_file_path):
with open(file_path, encoding="utf-8") as f:
data = json.load(f)

# Function to process each conversation
def process_conversation(item):
idx, conversation = item
results_for_conversation = []

# Process each question in the conversation
for question_item in tqdm(
conversation["qa"], desc=f"Processing questions for conversation {idx}", leave=False
):
if int(question_item.get("category", "")) == 5:
continue
result = self.process_question(question_item, idx)
results_for_conversation.append(result)

return idx, results_for_conversation

# Use multiprocessing to process the conversations in parallel
with Pool(processes=WORKERS) as pool:
results = list(
tqdm(
pool.imap(process_conversation, list(enumerate(data))),
total=len(data),
desc="Processing conversations",
)
)

# Reorganize results and store them in self.results
for idx, results_for_conversation in results:
self.results[f"locomo_exp_user_{idx}"] = results_for_conversation

# Save results to output file
with open(output_file_path, "w") as f:
json.dump(self.results, f, indent=4)


def main(version):
os.makedirs(f"results/locomo/openai-{version}/", exist_ok=True)
output_file_path = f"results/locomo/openai-{version}/openai_locomo_responses.json"
openai_predict = OpenAIPredict()
openai_predict.process_data_file("data/locomo/locomo10.json", output_file_path)


if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--version",
type=str,
default="default",
help="Version identifier for loading results (e.g., 1010)",
)
args = parser.parse_args()
version = args.version
main(version)
Loading
Loading