Deployment Change Log 61cefa2c5

Fix three_pl_dermapen autofill logic: set TESTING env var default to False; correct ForeignKey field name from three_pl_fulfilment to three_pl_order_fulfilment in model and usage; minor formatting and comment fixes (three_pl_dermapen app)

- Introduce centralized Redis lock keys and Valkey timestamp keys for ProductAvailability per-location updates in cached_dear.connector.dear_lock_keys.
- Refactor dear_cached_api.py to accumulate ProductAvailability create/update operations per location and commit them in a single atomic transaction under a short Redis lock.
- Use Valkey timestamp to detect and skip stale commits if a newer update has already been applied.
- Update hard allocation quantities within the same transaction, zeroing out non-observed allocations.
- Add detailed logging for lock acquisition, stale data detection, and commit results.
- Update documentation/locking.txt with example usage of optimistic cache update pattern using Redis locks and Valkey timestamps.
- Fix minor typo in aider_prompt.md.

Apps affected: cached_dear

Apps affected: cached_dear, dear_zoho_analytics, rest_api, shopify, three_pl, three_pl_aurora, three_pl_dermapen

Apps: cached_dear, three_pl

- In cached_dear/connector/dear_additional_logic.py, enhanced error messages in reauthorise_invoices to include current Dear sale JSON from API for better diagnostics.
- Minor message formatting fix in authorise_first_draft_invoice error logging.
- In cached_dear/dear_api_base/dear_base.py, updated get_sale method docstring to specify it returns the Dear Sale object from the API.

```
three_pl_dermapen: Refactor autofill order variable naming and improve logging config

- In three_pl_dermapen app, renamed all occurrences of 'autofill_order_row' to 'auto_fill_order_row' for consistency.
- Made DearInstanceSettings lookup case-insensitive using __iexact.
- Enhanced DermapenAutoFillOrderStatus enum to properly handle notification levels with a class-level map.
- Added detailed DEBUG-level logging configuration to local_settings.py for local development.
- Updated tests to reflect variable name changes.
```

- Introduce AutoFillNotificationLevel enum for log/email/attribute notifications.
- Enhance DermapenAutoFillOrderStatus with PICKED and PICKING_FAILED states and notification levels.
- Add fulfilment_nbr and link to three_pl_fulfilment in DermapenAutoFillOrder model.
- Centralize error/status handling in handle_autofill_status_update_and_error_reporting().
- Implement auto_pick_order() to run auto-pick, update DermapenAutoFillOrder status, and handle errors.
- Implement prepare_and_queue_three_pl_fulfilment() to create/refresh 3PL fulfilment row and enqueue it for 3PL export.
- Refactor task_process_dermapen_autofill_queue to use new helpers for picking and 3PL queueing, improving clarity and error handling.
- Add tests for auto_pick_order and prepare_and_queue_three_pl_fulfilment steps.

App: three_pl_dermapen

- Added JB_DISABLE_BUFFERING env var to pytest run config for three_pl_dermapen tests.
- In cached_dear app, fix ProductAvailability bulk_update to handle clearing stale flag without other field changes.
- In three_pl_dermapen business_logic/autofill_logic.py:
  - Introduce AutoFillErrorType enum and centralized error handler function.
  - Modify queue_dermapen_order_for_auto_fill_if_applicable to handle test monkeypatching.
  - Rename and enhance task_process_dermapen_autofill_queue task:
    - Accept optional row_ids to process specific orders.
    - Skip lock acquisition in testing mode.
    - Use DearCachedAPI instance reuse.
    - Add API_error exception handling inside sale lock block.
- In three_pl_dermapen tests, update test_one_order_dev to uppercase keys in monkeypatch dicts and call task_process_dermapen_autofill_queue with row_ids param.

Apps affected: three_pl_dermapen, cached_dear

# Conflicts:
#	ansible/includes/version.yml
#	dear_zoho_analytics/tasks.py

- Updated date range filtering in process_dear_sale_transaction to robustly parse and compare EffectiveDate as date objects instead of string comparison.
- Modified SQL JSONB date filter to cast EffectiveDate to date type for accurate date range filtering in create_intercompany_journals.
- These changes ensure correct handling of timestamps and date boundaries in transaction filtering.

- Removed unused imports (hashlib, logging, random, json) from tasks.py
- Cleaned up import statements for better readability
- Added @no_simultaneous_execution_no_retry decorator to refresh_za_product_movements_all_history task to prevent concurrent execution

- Enhanced autofill logic to handle active 3-PL locations using `three_pl_dispatch_mapping`.
- Introduced `active` flag inside dispatch configuration for location-level enablement and error email configuration.
- Updated Hoxton auto-process flow with feature flag `USE_V2_AUTOPROCESS` for conditional execution.
- Added defensive error handling and testing mode guard to improve queueing and processing logic.
- Implemented dummy 3PL backend for testing purposes without contacting external APIs.
- Enhanced logging, error handling, and location-based processing safeguards for better reliability.
- Improved typing and modular configuration for dispatch settings, ensuring easier extensibility.

# Conflicts:
#	ansible/includes/version.yml

- Added a check to avoid injecting banners on any request paths starting with "/admin" to prevent cluttering the Django admin interface.
- Added a guard in the new-version banner injection method to never show the banner while an update is pending.

Summary:
In the three_pl_dermapen app, modified the process_dermapen_autofill_queue function in autofill_logic.py to assign ProcessStatus.QUEUED to newly prepared fulfilment rows and clear their file_exchange_status and latest_result fields before saving. This ensures fulfilment rows are properly queued for 3PL transmission.

- In three_pl_dermapen, updated autofill_logic.py to create or refresh 3PL fulfilment row immediately after a pick is created, enabling immediate export to 3PL backend.
- Added error handling and logging around fulfilment row preparation.
- In tasks.py, added a comment noting that the task_dermapen_autopick_by_channel task will become redundant due to upcoming new auto fulfilment logic.

App: three_pl_dermapen

- Add a new BooleanField 'stale' to ProductAvailability model to mark rows not refreshed in the latest update.
- Modify update_product_availability_v2 to mark all rows for a location as stale before refresh, clear stale flag on updated/created rows, and remove immediate deletion of obsolete rows.
- Add sweep_stale_product_availability method in DearCachedAPI to delete rows still marked stale, guarded by a Redis lock.
- Create a new periodic Celery task (task_sweep_product_availability_stale_rows) to run the sweeper every 5 minutes for all active Dear instances.
- Update all queries on ProductAvailability to filter out stale=True rows to avoid using obsolete data.
- Fix minor typos and improve code style in cached_dear/connector/dear_cached_api.py.
- Add consistent lock key function for location fulfilment in cached_dear/connector/dear_lock_keys.py.
- Update imports and task scheduling in cached_dear/tasks_1300_tempfence.py to use task objects.
- Update various business logic and integration modules (three_pl, shopify, dear_zoho_analytics, rest_api) to exclude stale ProductAvailability rows.
- Add migration 0026_productavailability_stale.py to add the stale field to the database.

Apps affected: cached_dear, rest_api, dear_zoho_analytics, shopify, three_pl, three_pl_aurora

core: Promote heartbeat debug logs to info level for better visibility in production

- In core/celery_signals.py, changed heartbeat hook logging from debug to info to ensure visibility of task heartbeat start events and duplicate heartbeat warnings in production.
- In core/common_files/job_heartbeat.py, added info-level log when a heartbeat is published for a job.

Apps affected: core, cin7_sync (only file moved out, no code changes)

- Add DermapenAutoFillOrder model and status TESTING to track orders queued for auto-fill.
- Implement queue_dermapen_order_for_auto_fill_if_applicable() to enqueue eligible Dermapenworld orders based on location, RMA status, shipping country, and customer attribute.
- Add process_dermapen_autofill_queue Celery task to process queued orders, perform auto-picking, and create fulfilment rows.
- Move three_pl_dispatch_mapping and related settings to business_logic/definitions.py for reuse.
- Clean up redundant mapping in tasks.py and import from definitions.
- Add pytest tests for queuing logic and exclusion criteria.
- Add run configuration for pytest targeting the new test.

App affected: three_pl_dermapen

- cached_dear: Changed "List Redis locks" link text to "List locks" in index.html for clarity.
- core: Enhanced the update-pending banner message in show_banners_middleware.py to better explain job behavior during application updates.

three_pl, three_pl_dermapen: Improve 3PL order status handling and Dear shipment processing

- three_pl: Add guard to prevent regression of ACCEPTED status in fulfilment_rows update logic.
- three_pl: Add new ProcessStatus SHIPPED_IN_DEAR.
- three_pl_dermapen: Rename variable for clarity in task_create_dear_shipments.
- three_pl_dermapen: In find_and_process_hexspoor_status_updates, upgrade PENDING_3PL to ACCEPTED when Hexspoor acknowledges order.
- three_pl_dermapen: Adjust file_exchange_status and process_status updates for shipped orders; clarify shipment posting is handled by separate task.
- three_pl_dermapen: Fix file_exchange_status value for PICK AND PACK status to "ORDER_SENT_APPROVAL_RETURNED".

- core: Add update_pending.py to track update-pending flag with TTL and ETA
- core: Add views_update.py with signal_update_pending API endpoint to set flag, protected by secret
- core: Modify heavy_load_middleware.py to inject update-pending banner in HTML responses when flag active
- core: Add URL route for signal-update-pending API
- project: Add UPDATE_PENDING_SECRET config and load from env vars or common-secret-env.values
- amazon_eks: Replace common-secret-env.yaml with common-secret-env.values and update kustomization.yaml accordingly
- amazon_eks/helpers/tui_deployer.py: Notify running pods of update pending by calling signal-update-pending API before deployment
- three_pl_dermapen: Add new template three_pl_ref_row_status.html (unrelated to update-pending feature)

- cached_dear: Refactor PurgeCeleryQueuesView to purge Celery queues directly via Redis/Sentinel client instead of using celery_app.control.purge. Added robust Redis key deletion logic and improved logging.
- core: Rename heavy_load.py to heavy_load_middleware.py and update HeavyLoadBannerMiddleware with UI improvements including a dismiss button and adjusted banner positioning.
- project: Update middleware path in settings.py to reflect core.heavy_load_middleware.HeavyLoadBannerMiddleware.

Apps affected: cached_dear, core

- Introduced a global per-process throttle to ensure at least 1 second spacing between any two Zoho API calls within the same worker process, using a threading.Lock and monotonic timestamps.
- Replaced the fixed 1-second sleep before every Zoho API call with a dynamic wait based on the last call time to better respect Zoho rate limits.
- Removed per-row deletion logging in ZohoAnalytics deletion to reduce excessive log spam and avoid additional throttling delays.

- Moved CSS rules for navbar-nav.me-auto flex-wrap and overflow-x from main.css to inline style in base.html for better control.
- Re-enabled flex-wrap on .navbar-nav.me-auto at lg breakpoint to allow wrapping instead of overflow.
- Removed redundant overflow-x: auto on .navbar-collapse from main.css.
- Fixed malformed <title> tag in base.html.

- core/static/core/css/main.css: Added CSS rules to allow navbar menu wrapping on medium-large screens and horizontal scrolling on overflow, preventing menu spillover and ensuring link text doesn't break mid-word.
- core/templates/core/base.html: Changed navbar-expand-md to navbar-expand-lg for better navbar expansion breakpoint.
- core/tasks.py: Enhanced monitor_celery_queues task to send an "all-clear" email notification when all monitored Celery queues return below the alert threshold after a previous alert was sent, improving alert cycle handling. Added import for timezone.

core: enhance celery task handling and improve celery queue depths template

- Update celery_queue_depths.html to show separate "Ready" and "Unacked" columns with formatted timestamps.
- In core/celery_signals.py, add task_postrun signal handler to clean up leftover gevent greenlets after each task, preventing tasks from hanging due to lingering greenlets.
- Add graceful fallback if gevent is not installed.
- Refactor skip_disabled_periodic_tasks to raise Ignore for disabled periodic tasks without extra comments.

core: Improve Redis DB parsing and queue depth measurement in Celery monitoring

- Robustly parse Redis DB number from broker URL path, handling extra slashes or segments and falling back to settings.
- Enhance Celery queue depth measurement to try multiple Redis key patterns for ready and unacked lists, improving accuracy and visibility of unacknowledged tasks.

dear_zoho_analytics: Add 1-second throttle delay between Zoho API deletion requests to avoid rate limits

documentation: Rename and update webhook inbound docs with note on authentication method referencing generic_webhook_handler

- Created `delete_container_creating.sh` to list and delete Kubernetes pods stuck in 'ContainerCreating' status
- Supports checking all namespaces or a specific namespace
- Includes user confirmation before deletion to prevent accidental pod removal

Summary:
Added a helper function to clear the terminal screen on exit from the TUI deployer, ensuring a clean prompt even after errors or interrupts. Wrapped the main curses TUI call in a try-finally block to invoke this terminal clear function on exit.

- Add a new template cached_dear/celery_queue_depths.html to display Celery queue depths.
- Add links in cached_dear index.html menu for Celery queue depth view and spam test.
- Add CeleryQueueDepthsView and SpamCeleryQueueView in cached_dear/views_diagnostic.py for viewing queue depths and spamming main queue with dummy tasks.
- Add URL routes for these views in cached_dear/url_includes/urls_diagnostics.py.
- Add helper function get_celery_queue_depths in core/tasks.py to get Redis queue lengths.
- Add a simple 5-second sleep task sleep_test_5s in core/tasks.py used by spam view.
- Update amazon_eks/karpenter/ec2nodeclass.yaml to bump Amazon Linux 2023 AMI alias version.

Apps affected: cached_dear, core

- Refined logic to track whether an existing journal entry was actually updated.
- Updated counting of created and updated journal entries to reflect real changes.
- Added check to skip updates on fully posted journal entries while logging potential discrepancies.
- Ensured document_type changes trigger updates only when necessary.

Apps affected: cached_dear

Master pricing site hq

- Added a new POST action "Send Receipt Webhook" in dear_purchasing/dp_views.py to send selected warehouse receipt batches as a consolidated JSON payload to an external webhook endpoint.
- Implemented logging of webhook payloads and responses using JobMaster for audit and troubleshooting.
- Added a new submit button in dear_purchasing/templates/dear_purchasing/choose_batch_with_filter.html to trigger the webhook sending action.
- Created an OpenAPI spec docs/openapi/purchase_receipt_notification.yaml describing the webhook payload schema and endpoint.
- Added a PyCharm run configuration file .run/run_djcity.run.xml for running the Django project with specific environment variables.

App affected: dear_purchasing

Apps affected: `three_pl_dermapen`

- Refactor ResetFulfilmentRowStatusView to use FormView with proper form handling and permission checks.
- Add inline status update form to ThreePLRowStatus view with POST support and permission enforcement.
- Improve reset_status_confirmation.html to show a form for changing status instead of static confirmation.
- Enhance three_pl_row_status.html with better layout, Bootstrap styling, and inline status update form.
- Add _ResetProcessStatusForm for selecting new process status.
- Add detailed logging and user messages on status changes.
- Enforce permission checks on status updates.
- Clean up imports and remove redundant code.

revel_pos: Prevent duplicate PO creation from Dear restock transfers and fix PurchaseOrderAPI call

- In revel_pos/business_logic/restock_orders.py:
  - Added persistent tracking of processed Dear stock transfers using KeyValueJson to avoid duplicate PO creation.
  - Updated Dear Stock-Transfer "Reference" field with a sentinel string after successful PO submission to mark as exported.
  - Added logging for duplicate skips and update failures.
- In revel_pos/revel_connector.py:
  - Fixed PurchaseOrderAPI.api_request call to use named 'endpoint' parameter for clarity.
- In revel_pos/tasks.py:
  - Commented out redundant old task_create_revel_POs_from_dear_restocks task code to reduce clutter.
- In revel_pos/tests/test_revel_po_logic.py:
  - Removed unused import of task_create_revel_POs_from_dear_restocks.

These changes improve robustness by preventing duplicate PO submissions from the same Dear restock transfer and clean up legacy code.

- three_pl: Add info icon tooltip to stock_cover_rating and process_status columns in ThreePL_order_fulfilmentsTable.
- three_pl_dermapen:
  - Add tooltips to filter labels (warehouse_name, order_age) in ThreePLFulfilmentsFilter and persist order_age filter in session.
  - Add tooltip to three_PL_fulfilment_id column in ThreePL_order_fulfilmentsTable_dermapen.
  - Improve error message logging in task_send_queued_orders to use latest_result from DB if available.
  - Remove commented-out RuntimeError raise in task_dermapen_autopick_by_channel.
  - Add missing import of mark_safe in views_dermapen.py.

- Enhanced exception handling in send_one_fulfilment_row to log a clearer message.
- Use row.latest_result if available for error details, otherwise fallback to exception string.
- Updated log message to include error details and specify Hexspoor send failure.

- core/tasks.py: Improve evict_stale_gevent_jobs to only target gevent queue tasks and revoke them on specific workers; update default age_hours to 3.
- three_pl/business_logic/fulfilment_rows.py: Change job log statuses from ERROR/COMPLETED to INFO when reporting completion with or without errors.
- three_pl_dermapen/tasks.py: Change various job log statuses from OK/COMPLETED to INFO or OK for consistency and clarity.

- Added 'OPTIONS': {'server_side_cursors': False} to default and dear_analytics database settings.
- Introduced global setting DISABLE_SERVER_SIDE_CURSORS = True to disable server-side cursors on all DB connections.
- This prevents errors caused by PgBouncer's transaction-pooling mode dropping session state.

```
dear_zoho_analytics, three_pl: Fix DB cursor handling and improve 3PL row filtering logic

- dear_zoho_analytics: Materialize ProductAvailability queryset to list before further DB writes to avoid psycopg2 InvalidCursorName errors due to server-side cursor closure.
- three_pl: Refine filtering in update_rows_pass_2 to include Netherlands rows with Cancelled status and pending 3PL process status; add trimming and case normalization for warehouse_name and latest_result checks.
```

Apps: three_pl, three_pl_dermapen

- Updated kustomization_pre_subst.yaml in multiple amazon_eks overlays (e.g., 1300tempfence, aurora-lites, ausab, bandr, barbell, cantontea, chefworks, cultiver, curve, djcity, equipmed, glenmar, graintec, interioricons, kelato, littlelinen, livingbydesign, loam, marle, onemile, prestige, prp, purefood, retrojan, satara, sprint, statusanxiety, thesaltyfox, tilecloud, vv, xysense)
- Added a patch to overwrite the stringData of the pgbouncer-auth Secret with userlist.txt entries using variables (e.g., ${short_hostname}, ${django_api_sync_db_password}, ${analytics_db_password}, ${omni_analytics_db_password}) that will be substituted by the deployment script
- Removed redundant empty default literals from tilecloud overlay's site-secret generator

App: amazon_eks

# Conflicts:
#	ansible/includes/version.yml
#	three_pl/tasks_djcity.py
#	zoho_crm/zoho_crm_business_logic/dear_to_zoho_quote_sync.py

App: amazon_eks

- In core app, enhance database diagnostics by adding a direct connection test to the PostgreSQL server using psycopg2, bypassing PgBouncer.
- This extra check uses the same credentials as Django and respects any POSTGRES_HOST/PORT environment overrides.
- Minor comment fix in amazon_eks/helpers/configure_new_version.sh (typo in comment).

Update Zoho 3PL stock analysis progress logging and fix quote sync test formatting

- three_pl_dermapen: Added detailed job progress logging with job_score updates during Zoho 3PL stock analysis refresh, including start, row processing progress, upload completion, and final completion status.
- zoho_crm: Fixed string literal formatting in dear_to_zoho_quote_sync test to correctly escape backslash.
- amazon_eks: Updated test-dear site-secret.yaml to add md5 hash entries for passwords.

Apps affected: three_pl_dermapen, zoho_crm, amazon_eks

# Conflicts:
#	ansible/includes/version.yml

core, three_pl: Add acks_late=False to Celery tasks and improve logging

- core: Added acks_late=False to several Celery tasks to acknowledge tasks before running, improving task handling and reliability.
- core: Added @no_simultaneous_execution_no_retry decorator to job_log_delete and other tasks to prevent concurrent execution.
- three_pl: Added acks_late=False to Celery tasks in tasks_djcity.py and improved job logging messages for better traceability.

core: Add publish_to_websocket=False to job.log_message in task decorators and task_cleanup_stale_locks; improve webhook task processing locking

- Suppress websocket publishing for certain job.log_message calls in no_simultaneous_execution_no_retry decorator and task_cleanup_stale_locks task to reduce noise.
- In process_due_webhook_tasks, mark selected webhook tasks as PROCESSING and update next_attempt_time to prevent duplicate processing.
- Add @no_simultaneous_execution_no_retry decorator to task_cleanup_stale_locks to prevent concurrent execution.

# Conflicts:
#	ansible/includes/version.yml

App: cached_dear