I’m using LLaMA 3.1 to generate optimized JavaScript code variants by following a prompt that enforces data minimization principles. My goal is for the model to either optimize or confirm that no changes are necessary to the provided JavaScript code. Despite careful setup, the model’s responses often don’t accurately reflect the instructions given. Specifically, it always fails to optimize as requested.
Here’s a simplified version of my code:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", torch_dtype=torch.float16).to(device)
torch.backends.cudnn.benchmark = True
def generate_variants(filter_code, n_variants=3):
variants = []
text = f"""
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
The following definition refers to data minimization:
'Data minimization is a principle restricting data collection to what is necessary in relation to the purposes for which they are processed.'
<|eot_id|><|start_header_id|>user<|end_header_id|>
Please optimize the provided JavaScript code with the following instructions:
- If excessive anonymization is applied without clear utility, reduce the level of data anonymization to retain only the necessary level.
- Eliminate unnecessary API function calls, keeping only those essential for minimal and efficient data processing.
- Remove any unnecessary data attributes, ensuring that only essential data attributes are collected, processed, and stored.
If the code already complies with data minimization, please add a comment '// No changes needed' to indicate that no modifications are required.
Return only the JavaScript code, without any additional explanations, comments, or introductory text.
Mock the code to make it run.
Here is the code to optimize:
{filter_code}
<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
"""
for i in range(n_variants):
seed_value = random.randint(0, 10000)
torch.manual_seed(seed_value)
random.seed(seed_value)
input_ids = tokenizer(text, return_tensors="pt").input_ids.to(device)
generated_ids = model.generate(
input_ids,
max_length=1000,
temperature=0.1,
top_k=5,
top_p=0.8
)
response = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
cleaned_response = response[len(text):].strip()
if "// No changes needed" in cleaned_response:
cleaned_response = f"{filter_code} // No changes needed"
variants.append(cleaned_response)
return variants
I’ve set specific parameters, and I’m using a fixed prompt and I’m not allowed to adjust the prompt text or these generation parameters.
This is an e.g.of one of the code to optimize:
const GoogleCalendar = {
newEventAdded: {
Where: "[some street address]",
Starts: "9:00 AM",
Ends: "10:00 AM"
},
addDetailedEvent: {
skip: () => console.log("Event skipped."),
setDescription: (description) => console.log(`Description set: ${description}`),
setAllDay: (isAllDay) => console.log(`All-day set: ${isAllDay}`),
setStartTime: (startTime) => console.log(`Start time set: ${startTime}`),
setEndTime: (endTime) => console.log(`End time set: ${endTime}`)
}
};
if (GoogleCalendar.newEventAdded.Where.indexOf("[some street address]") < 0) {
GoogleCalendar.addDetailedEvent.skip();
} else {
GoogleCalendar.addDetailedEvent.setDescription("In the office from "
+ GoogleCalendar.newEventAdded.Starts
+ " to " + GoogleCalendar.newEventAdded.Ends);
GoogleCalendar.addDetailedEvent.setAllDay("true");
GoogleCalendar.addDetailedEvent.setStartTime(GoogleCalendar.newEventAdded.Starts);
GoogleCalendar.addDetailedEvent.setEndTime(GoogleCalendar.newEventAdded.Ends);
}
The output is always either a wrong generated code or always says that No changes needed even if clearly it needs it
Is there something wrong with my prompt template?