Familia v2.0.0-pre Series Technical Reference
Familia is a Ruby ORM for Valkey/Redis providing object mapping, relationships, and advanced features like encryption, connection pooling, and permission systems. This technical reference covers the major classes, methods, and usage patterns introduced in the v2.0.0-pre series.
For conceptual understanding and getting started with Familia, see the Overview Guide. This document provides detailed implementation patterns and advanced configuration options.
Core Architecture
Base Classes
Familia::Horreum - Primary ORM Base Class
The main base class for Valkey/Redis-backed objects, similar to ActiveRecord models.
class User < Familia::Horreum
# Basic field definitions
field :name, :email, :created_at
# Valkey/Redis data types as instance variables
list :sessions # Valkey/Redis list
set :tags # Valkey/Redis set
zset :scores # Valkey/Redis sorted set
hashkey :settings # Valkey/Redis hash
end
Key Methods:
save- Persist object to Valkey/Redissave_if_not_exists- Conditional persistence, returns false if existssave_if_not_exists!- Conditional persistence, raises RecordExistsError if existsload- Load object from Valkey/Redisexists?- Check if object exists in Valkey/Redisdestroy- Remove object from Valkey/Redis
Familia::DataType - Valkey/Redis Data Type Wrapper
Base class for Valkey/Redis data type implementations.
Registered Types:
String- Valkey/Redis stringsList- Valkey/Redis listsSet- Valkey/Redis setsSortedSet- Valkey/Redis sorted setsHashKey- Valkey/Redis hashesCounter- Atomic countersLock- Distributed locks
Feature System
Feature Architecture
Modular system for extending Horreum classes with reusable functionality.
class Customer < Familia::Horreum
feature :expiration # TTL management
feature :safe_dump # API-safe serialization
feature :encrypted_fields # Field encryption
feature :transient_fields # Non-persistent fields
feature :relationships # Object relationships
field :name, :email
encrypted_field :api_key
transient_field :password
end
Built-in Features
1. Expiration Feature
TTL (Time To Live) management with cascading expiration.
class Session < Familia::Horreum
feature :expiration
field :user_id, :token
default_expiration 24.hours
# Cascade expiration to related objects
cascade_expiration_to :user_activity
end
session = Session.new(user_id: 123, token: "abc123")
session.save
session.expire_in(1.hour) # UnsortedSet custom expiration
session.ttl # Check remaining time
2. SafeDump Feature
API-safe serialization excluding sensitive fields.
class User < Familia::Horreum
feature :safe_dump
field :name, :email, :password_hash
safe_dump_field :name, :email # Only these fields in safe_dump
end
user = User.new(name: "Alice", email: "alice@example.com", password_hash: "secret")
user.safe_dump # => {"name" => "Alice", "email" => "alice@example.com"}
3. Encrypted Fields Feature
Transparent field-level encryption with multiple providers.
# Configuration
Familia.configure do |config|
config.encryption_keys = {
v1: ENV['FAMILIA_ENCRYPTION_KEY'],
v2: ENV['FAMILIA_ENCRYPTION_KEY_V2']
}
config.current_key_version = :v2
end
class Vault < Familia::Horreum
feature :encrypted_fields
field :name # Plaintext
encrypted_field :secret_key # Encrypted with XChaCha20-Poly1305
encrypted_field :api_token # Field-specific key derivation
encrypted_field :private_data # Transparent access
end
vault = Vault.new(
name: "Production Secrets",
secret_key: "super-secret-123",
api_token: "sk-1234567890"
)
vault.save
# Transparent access - automatically encrypted/decrypted
vault.secret_key # => "super-secret-123" (decrypted on access)
Encryption Providers:
- XChaCha20-Poly1305 (preferred) - Requires
rbnaclgem - AES-256-GCM (fallback) - Uses OpenSSL, no dependencies
Advanced Security Implementation:
# Multiple encryption providers with fallback
class CriticalData < Familia::Horreum
feature :encrypted_fields
encrypted_field :credit_card
encrypted_field :ssn
encrypted_field :notes
end
# Key versioning and rotation
Familia.configure do |config|
config.encryption_keys = {
v1: ENV['OLD_KEY'], # Legacy key
v2: ENV['CURRENT_KEY'], # Current key
v3: ENV['NEW_KEY'] # New key for rotation
}
config.current_key_version = :v2
config.encryption_personalization = 'MyApp-2024' # XChaCha20 BLAKE2b domain separation (<= 16 bytes)
config.encryption_hkdf_salt = 'MyApp-2024' # AES-GCM HKDF salt domain separation (any length)
# When rotating either value, keep prior values for backward-compatible decryption:
# config.encryption_personalization_history = ['MyApp-2023']
# config.encryption_hkdf_salt_history = ['MyApp-2023']
end
# Operations with encrypted fields
record = CriticalData.new(
credit_card: "4111-1111-1111-1234",
ssn: "123-45-6789",
notes: "Customer notes"
)
record.save
# Verify encryption status
puts "Record #{record.identifier}: #{status}"
# => {credit_card: {encrypted: true, algorithm: "xchacha20poly1305", cleared: false}}
ConcealedString Security Features:
# Automatic protection against accidental exposure
user = CriticalData.load("user123")
# Safe operations
user.credit_card.class # => ConcealedString
user.credit_card.to_s # => "[CONCEALED]"
user.credit_card.inspect # => "[CONCEALED]"
user.credit_card.to_json # => "\"[CONCEALED]\""
# Explicit access when needed
actual_card = user.credit_card.reveal # => "4111-1111-1111-1234"
# Logging safety
Rails.logger.info "Processing card: #{user.credit_card}"
# => "Processing card: [CONCEALED]" (safe for logs)
# JSON serialization safety
user_data = user.to_json
# All encrypted fields show as "[CONCEALED]" in JSON
4. Transient Fields Feature
Non-persistent fields with memory-safe handling.
class LoginForm < Familia::Horreum
feature :transient_fields
field :username # Persistent
transient_field :password # Never stored in Redis
transient_field :csrf_token # Runtime only
end
form = LoginForm.new(username: "alice", password: "secret123")
form.save # Only saves 'username', password is transient
form.password.class # => Familia::Features::TransientFields::RedactedString
form.password.to_s # => "[REDACTED]" (safe for logging)
form.password.reveal # => "secret123" (explicit access)
RedactedString - Security wrapper preventing accidental exposure:
to_sreturns "[REDACTED]"inspectreturns "[REDACTED]"revealmethod for explicit access- Safe for logging and serialization
5. Relationships Feature
Comprehensive object relationship system with automatic management, clean Ruby-idiomatic syntax, and simplified method generation.
class Customer < Familia::Horreum
feature :relationships
identifier_field :custid
field :custid, :name, :email
# Collections for storing related object IDs
set :domains # Simple set
zset :activity # Scored/sorted collection
# Class-level indexed lookups (automatically managed on save/destroy)
class_indexed_by :email, :email_lookup
# Class-level tracking with scoring (automatically managed on save/destroy)
class_participates_in :all_customers, score: :created_at
end
class Domain < Familia::Horreum
feature :relationships
identifier_field :domain_id
field :domain_id, :name, :status
# Bidirectional membership with clean << operator support
participates_in Customer, :domains
# Relationship-scoped indexing (per-customer domain lookups)
indexed_by :name, :domain_index, target: Customer
# Class-level conditional tracking with lambda scoring
class_participates_in :active_domains,
score: -> { status == 'active' ? Familia.now.to_i : 0 }
end
Relationship Operations with Automatic Management:
# Create and save objects (automatic indexing and tracking)
customer = Customer.new(custid: "cust123", name: "Acme Corp", email: "admin@acme.com")
customer.save # Automatically adds to email_lookup and all_customers
domain = Domain.new(domain_id: "dom456", name: "acme.com", status: "active")
domain.save # Automatically adds to active_domains
# Establish relationships using clean Ruby-like << operator
customer.domains << domain # Clean, Ruby-idiomatic collection syntax
# Query relationships
domain.in_customer_domains?(customer.custid) # => true
customer.domains.member?(domain.identifier) # => true
# O(1) indexed lookups (automatic management - no manual calls needed)
found_id = Customer.email_lookup.get("admin@acme.com")
found_customer = Customer.find_by_email("admin@acme.com") # Convenience method
# Relationship-scoped lookups
customer_domain = customer.find_by_name("acme.com") # Find within customer
# Class-level tracking queries
recent_customers = Customer.all_customers.range_by_score(
(Familia.now - 24.hours).to_i, '+inf'
)
active_domains = Domain.active_domains.members
Key Features:
- Automatic Management: Objects automatically added/removed from class-level collections on save/destroy
- Clean Syntax: Collections support Ruby-like
customer.domains << domainsyntax - Simplified Methods: No complex "global" terminology - uses clear
class_prefixes - Performance: O(1) hash lookups and efficient sorted set operations
- Flexibility: Supports class-level and relationship-scoped indexing patterns
6. Object Identifier Feature
Automatic generation of unique object identifiers with configurable strategies.
Default generator is :uuid_v7 (UUID version 7 with embedded timestamp).
class Document < Familia::Horreum
feature :object_identifier # Uses default :uuid_v7
field :title, :content, :created_at
end
class Session < Familia::Horreum
feature :object_identifier, generator: :hex, length: 16
field :user_id, :data, :expires_at
end
class ApiKey < Familia::Horreum
feature :object_identifier, generator: :custom
field :name, :permissions, :created_at
# Custom generator implementation
def self.generate_identifier
"ak_#{SecureRandom.alphanumeric(32)}"
end
end
Generator Types:
:uuid_v7- UUID version 7 with embedded timestamp (default, 36 characters):uuid_v4- Standard UUID v4 format (36 characters):hex- High-entropy hexadecimal strings (256-bit via SecureIdentifier)- Proc/Lambda - Custom generation logic provided as a callable
Technical Implementation:
# Auto-generated on object creation
doc = Document.create(title: "My Document")
doc.objid # => "01234567-89ab-7def-8000-123456789abc" # UUID v7 format
session = Session.create(user_id: "123")
session.objid # => "a1b2c3d4e5f67890"
# Custom identifier with proc
api_key = ApiKey.create(name: "Production API")
api_key.objid # => "ak_Xy9ZaBcD3fG8HjKlMnOpQrStUvWxYz12"
# Feature options per-class isolation
Document.(:object_identifier)
#=> {generator: :uuid_v7}
7. External Identifier Feature
Derives deterministic external identifiers from object identifiers.
class ExternalUser < Familia::Horreum
feature :external_identifier
identifier_field :id
field :id, :name
# External ID is automatically derived from objid
# Format: 'ext_' + base36(truncated_hash(objid))
end
class APIKey < Familia::Horreum
feature :external_identifier, format: 'api-%{id}'
field :name, :permissions
end
External ID Management:
# External ID is deterministically derived from objid
user = ExternalUser.new(name: "John Doe")
user.save
user.objid # => "01234567-89ab-7def-8000-123456789abc"
user.extid # => "ext_abc123def456ghi789" # Deterministic from objid
# Same objid always produces same extid
user2 = ExternalUser.new(objid: user.objid, name: "John Doe")
user2.extid # => "ext_abc123def456ghi789" # Identical
# Custom format with APIKey
key = APIKey.new(name: "Production")
key.extid # => "api-xyz789abc123"
8. Quantization Feature
Time-based data bucketing for analytics and caching.
class DailyMetric < Familia::Horreum
feature :quantization
identifier_field :metric_key
field :metric_key, :bucket_timestamp, :value_count, :sum_value
# Example: Basic time quantization
string :counter, default_expiration: 1.day, quantize: [10.minutes, '%H:%M']
end
Basic Usage:
# Create metric with quantized timestamp
metric = DailyMetric.new(metric_key: "page_views")
metric.counter.increment
# Time-based data grouping occurs automatically
# All data within the same 10-minute window shares the same key
Key Benefits:
- Time Bucketing: Group time-based data into configurable intervals
- Reduced Storage: Aggregate similar data points to optimize memory usage
- Analytics Ready: Perfect for dashboards and time-series data visualization
Advanced Feature System Architecture
Feature Autoloader for Project Organization
Automatically load features from directory structure.
# app/models/customer.rb - Main model file
class Customer < Familia::Horreum
include Familia::Features::Autoloader
# Automatically loads all .rb files from app/models/customer/*.rb
# Core model definition
identifier_field :custid
field :custid, :name, :email, :created_at
end
# app/models/customer/notifications.rb
class Customer < Familia::Horreum
def send_welcome_email
NotificationService.send_template(
email: email,
template: 'customer_welcome',
variables: { name: name, custid: custid }
)
end
def send_invoice_reminder(invoice_id)
NotificationService.send_template(
email: email,
template: 'invoice_reminder',
variables: { invoice_id: invoice_id }
)
end
end
# app/models/customer/analytics.rb
class Customer < Familia::Horreum
# Analytics methods added to Customer
def track_activity(activity_type, = {})
activity_data = {
custid: custid,
activity: activity_type,
timestamp: Familia.now.to_i,
metadata:
}
# Store in customer's activity stream
activities.unshift(activity_data.to_json)
activities.trim(0, 999) # Keep last 1000 activities
end
def recent_activities(limit = 10)
activities.range(0, limit - 1).map { |json| JSON.parse(json) }
end
end
Feature Dependencies
Features can declare dependencies that are automatically resolved.
# External identifier depends on object_identifier
class User < Familia::Horreum
feature :external_identifier # Automatically includes :object_identifier
field :name
end
# Feature dependency resolution
Familia::Base.add_feature ExternalIdentifier, :external_identifier, depends_on: [:object_identifier]
# When external_identifier is included, object_identifier is automatically loaded first
end
Per-Class Feature Registration
Register custom features for specific model classes with ancestry chain lookup.
# Define a custom feature module
module CustomerAnalytics
def track_purchase(amount)
purchases.increment(amount)
end
end
# Register feature only for Customer and its subclasses
Customer.add_feature CustomerAnalytics, :customer_analytics
class Customer < Familia::Horreum
feature :customer_analytics # Available via Customer's registry
end
class PremiumCustomer < Customer
feature :customer_analytics # Inherited via ancestry chain
end
class Session < Familia::Horreum
# feature :customer_analytics # Not available - would raise error
end
Benefits:
- Features can have the same name across different model hierarchies
- Natural inheritance through Ruby's class hierarchy
- Better namespace management for large applications
Per-Class Feature Configuration Isolation
Each class maintains independent feature options.
class PrimaryCache < Familia::Horreum
feature :expiration
feature :object_identifier, generator: :uuid_v7
field :cache_key, :value, :hit_count
default_expiration 24.hours
end
class SecondaryCache < Familia::Horreum
feature :expiration
feature :object_identifier, generator: :hex # Different generator
field :cache_key, :backup_value, :backup_timestamp
default_expiration 7.days
end
# Feature options are completely isolated
PrimaryCache.(:object_identifier)
#=> {generator: :uuid_v7}
SecondaryCache.(:object_identifier)
#=> {generator: :hex}
Runtime Feature Checking
Check which features are enabled on a class.
class SecureModel < Familia::Horreum
feature :expiration
feature :encrypted_fields
feature :safe_dump
field :name, :status
encrypted_field :api_key
safe_dump_field :name
end
# Check enabled features
SecureModel.features_enabled
#=> [:expiration, :encrypted_fields, :safe_dump]
# Check feature options
SecureModel.(:encrypted_fields)
#=> {} # Default options
# Each class tracks its own features
class BasicModel < Familia::Horreum
feature :expiration
field :name
end
BasicModel.features_enabled
#=> [:expiration]
Connection Management
Connection Provider Pattern
Flexible connection pooling with provider-based architecture.
# Basic Valkey/Redis connection
Familia.configure do |config|
config.uri = "redis://localhost:6379/0"
end
# Custom connection provider with pooling
require 'connection_pool'
POOLS = {}
POOLS_MUTEX = Mutex.new
Familia.connection_provider = lambda do |uri|
# Provider MUST return a connection already on the correct database.
# Build each pool ONCE and hand back a wrapper (see the contract below).
POOLS_MUTEX.synchronize do
POOLS[uri] ||= ConnectionPool::Wrapper.new(size: 10, timeout: 5) do
Redis.new(url: uri)
end
end
end
Provider contract
Familia calls the provider every time it needs a client and never hands the connection back — there is no check-in hook. The provider must therefore return a client that stays exclusively usable by the caller without a matching check-in, and it must not build a new pool per call.
Do not do this:
Familia.connection_provider = lambda do |uri|
pool = ConnectionPool.new { Redis.new(url: uri) } # a new pool every call
pool.with { |conn| conn } # checked back in already
end
ConnectionPool#with checks the connection back in the moment its block
returns, so the client handed to Familia is simultaneously available to every
other checkout — the pool reports it as free while Familia is still using it:
pool = ConnectionPool.new(size: 5) { Redis.new }
conn = pool.with { |c| c }
pool.available # => 5 of 5 — nothing is checked out, yet `conn` is in use
Under concurrency two threads then issue commands on one connection, and per-connection state (MULTI, WATCH, SELECT, SUBSCRIBE) crosses callers.
Use ConnectionPool::Wrapper (alias ConnectionPool.wrap) instead. It
proxies each command through pool.with, so a connection is checked out for
exactly the duration of that command and checked back in afterwards.
ConnectionPool's checkout is reentrant per fiber, so a MULTI/EXEC, a pipeline,
or a WATCH-guarded transaction still runs entirely on one connection.
Or check out explicitly when a request should pin one connection for its whole lifetime — then the checkout is yours to release at the boundary:
Familia.connection_provider = ->(uri) { POOLS.fetch(uri).checkout }
# Rack middleware / job wrapper
def call(env)
@app.call(env)
ensure
POOLS.each_value { |pool| pool.checkin(force: true) }
end
Without that ensure, a bare pool.checkout in the provider leaks a
connection on every call and the pool exhausts.
When a provider is configured, Familia.with_connection_scope steps aside:
ProviderConnectionHandler runs before ScopedConnectionHandler, so the scope
holds nothing for the lookups the provider serves.
Connection Scopes
Familia.with_connection_scope holds one connection per URI for the duration
of a block and closes those connections when the block exits.
Without a provider, every dbclient lookup that reaches
CreateConnectionHandler returns a new Redis client that nothing closes. A
scope gives those lookups an owner. While the block runs, each lookup in the
calling fiber that would otherwise reach CreateConnectionHandler resolves to
the client the scope holds for that connection configuration, created on
first use with Familia.create_dbclient. When the block exits, by return or by
exception, the scope closes every client it created. Resolution outside a
scope is unchanged.
Familia.with_connection_scope do |client|
client.ping # the scope's client for Familia.uri
customer = Customer.load(custid) # resolves to the same client
customer.plan = 'premium'
customer.save
end
# The clients the scope created are closed.
Yielded client. The block receives the scope's client for uri
(Familia.uri when nil; an Integer selects a logical database on the default
server), resolved through Familia.dbclient. Inside an open transaction or
pipeline the yielded client is that transaction or pipeline connection.
Nesting. A with_connection_scope opened inside an active scope of the
same fiber joins it. Nothing is acquired or released until the outer block
exits.
Transactions and pipelines. ScopedConnectionHandler declares
supports transaction: true, pipelined: true. transaction, pipelined and
atomic_write inside the block resolve their connection through the chain, so
the MULTI/EXEC or pipeline runs on the scope's client for its URI. A
WATCH-guarded atomic_write resolves the connection once and issues WATCH and
MULTI/EXEC on it. While a transaction or pipeline block is open,
FiberTransactionHandler or FiberPipelineHandler serves that connection
ahead of the scope.
WATCH windows. EXEC, DISCARD and UNWATCH on a connection clear every key
that connection watches. Between the WATCH and the EXEC of a WATCH-guarded
transaction (atomic_write and Familia.atomic_write with watch_keys:,
save_if_not_exists!, Horreum#watch), the scope does not hand the watching
client to other lookups: they get a second client for the same URI, which the
scope holds, reuses and closes like the first. A record saved from a
pre_check therefore commits on its own connection and cannot clear the
outer WATCH. The watched MULTI itself runs on the connection the transaction
resolved. Horreum#watch yields the watching connection, and the MULTI/EXEC,
UNWATCH and DISCARD must be issued on it:
model.watch { |conn| conn.multi { |m| ... } }. Inside the block,
model.transaction, model.unwatch and model.discard resolve to the second
client, so a MULTI/EXEC sent through them is not guarded by the WATCH.
Ownership. A scope belongs to the fiber that opened it. A new fiber starts
with a copy of the opener's fiber storage (Ruby's documentation for
Fiber.new: "If the storage is unspecified, the default is to inherit a copy
of the storage from the current fiber."), and a new thread's root fiber does
the same on Ruby 3.3 (observed; the Thread documentation does not state it).
A thread or fiber spawned inside the block therefore sees the same
Fiber[:familia_connection_scope] value, but Scope.current compares the
scope's owner with Fiber.current and returns nil for them. Those threads and
fibers resolve connections as if no scope were open and open their own scope
when they need one. A direct Scope#client call from another fiber raises
Familia::Problem.
URI identity. Clients are keyed by URI#conf, the connection parameters
the URI resolves to (host, port, database, TLS, password and query options such
as timeout). redis://host:6379 and redis://host:6379/0 share one client; a
different logical database or server gets its own.
Precedence. A fiber-local connection set with Familia.fiber_connection=,
a configured connection_provider and a Horreum class @dbclient run before
the scope in their chains, so the scope holds nothing for the lookups they
serve. A standalone DataType instance's @dbclient is checked by
StandaloneConnectionHandler before it calls Familia.dbclient.
Reset. Familia.reconnect! and the setters that bump
Familia.middleware_version (enable_database_logging=,
enable_database_counter=, connection_provider=) do not touch an open
scope. The scope compares the version on its next lookup and retires the
clients it created under the old version: new lookups get fresh clients, and
the retired ones stay open until the block exits, so a handle the block
already holds, or a WATCH or MULTI in flight on it, keeps its connection.
Scope#clients and Scope#size include retired clients. Closing them at once
would let the Redis object reconnect silently on its next command, which
drops a WATCH and lets the following EXEC commit over a concurrent write.
Limits. A client reference kept past the block reconnects on its next
command as an unowned connection that nothing closes; Familia::Migration
instances memoize dbclient, so run a migration inside one scope rather than
across several. Enumerator#next and #peek run the enumerator in a fiber of
its own, so lookups made there resolve as if no scope were open. The scope's
client for a URI is shared by every lookup for that URI in the block, so
per-connection state set on it (SELECT, CLIENT SETNAME, SUBSCRIBE) is visible
to the next lookup.
Fork. A scope that spans fork keeps its Redis objects in the child.
What happens to an inherited socket is decided by redis-client, not by Familia.
redis-client 0.29.0, lib/redis_client.rb, in ensure_connected:
close if !config.inherit_socket && @pid != PIDCache.pid
Opening the scope after the fork avoids relying on that behaviour.
Inspection and release. Familia.current_connection_scope returns the
scope the current fiber has open, or nil. Familia::Connection::Scope exposes
owner (the opening fiber), clients (the held clients: retired ones first,
then the current ones grouped by connection configuration in the order each
was first used, each group in acquisition order), size, active?, released? and release!. release! closes every
held client and marks the scope released; a later Scope#client call raises
Familia::Problem. Called anywhere in the block, a joined inner block
included, it releases the scope for the rest of the outer block: plain
lookups after it resolve as if no scope were open, and a
with_connection_scope opened after it opens a new scope of its own, which
closes its clients when that inner block exits. It is safe to call more than
once; the block's own exit calls it again as a no-op. Only the owning fiber
may call it; any other fiber gets Familia::Problem. If a client's close
raises something other than a StandardError, such as an Interrupt, the
scope still closes the other clients and then raises it.
Horreum classes and DataType instances. Familia::Connection::Behavior
adds with_connection_scope(uri = nil) to Horreum classes and DataType
instances. It opens the same scope and yields the client resolved through the
receiver's own chain, so a class-level @dbclient or a logical_database
setting is honoured.
Customer.with_connection_scope do
custids.each do |custid|
customer = Customer.load(custid)
customer.plan = 'premium'
customer.save
end
end
Rack middleware
class FamiliaConnectionScope
def initialize(app)
@app = app
end
def call(env)
Familia.with_connection_scope { @app.call(env) }
end
end
use FamiliaConnectionScope
Each request holds one client per URI it touches and closes them when call
returns. Commands issued after call returns, for example by a streaming body,
run outside the scope.
Job or script
Familia.with_connection_scope do
custids.each do |custid|
customer = Customer.load(custid)
next unless customer
customer.plan = 'premium'
customer.save
end
end
Without the scope, every dbclient lookup that reaches the chain's fallback
returns a new client.
Multi-Database Support
Configure different logical databases for different models.
class User < Familia::Horreum
logical_database 0 # Use database 0
field :name, :email
end
class Session < Familia::Horreum
logical_database 1 # Use database 1
field :user_id, :token
end
class Cache < Familia::Horreum
logical_database 2 # Use database 2
field :key, :value
end
Testing and Debugging
Database Command Logging
Monitor all Redis commands with DatabaseLogger middleware.
# Enable command logging (middleware registered automatically)
Familia.enable_database_logging = true
# Capture commands in tests
commands = DatabaseLogger.capture_commands do
user = User.create(name: "Test User")
user.save
end
puts commands.first.command # => "SET user:123 {...}"
puts commands.first.μs # => 567 (microseconds)
# Enable sampling for production
DatabaseLogger.sample_rate = 0.01 # Log 1% of commands
# Structured logging format
DatabaseLogger.structured_logging = true
# => "Redis command cmd=SET args=[key, value] duration_ms=0.42 db=0"
Debug Mode
Enable comprehensive debugging output.
# Via environment variable
ENV['FAMILIA_DEBUG'] = '1'
ENV['FAMILIA_TRACE'] = '1'
# Via configuration
Familia.configure do |config|
config.debug = true
end
# Check database contents
Familia.dbclient.keys('user:*')
# Trace specific operations
Familia.trace :LOAD, redis_client, "user:123", "from cache"
Connection Chain Debugging
Understand connection resolution order.
# Connection resolution order (Familia.dbclient and Horreum class chains):
# 1. Active pipeline or transaction connection
# (Fiber[:familia_pipeline], Fiber[:familia_transaction])
# 2. Fiber-local connection (Familia.fiber_connection=)
# 3. connection_provider
# 4. Class-level @dbclient (Horreum classes only)
# 5. Open connection scope (Familia.with_connection_scope)
# 6. Fresh connection (CreateConnectionHandler)
#
# A DataType instance runs steps 1 to 3, then delegates to its parent's chain
# or, when standalone, uses its own @dbclient if set and otherwise
# Familia.dbclient.
class DebugModel < Familia::Horreum
logical_database 3
field :name
end
model = DebugModel.new(name: "test")
# Uses database 3 from logical_database
DebugModel.dbclient = custom_connection
# Class-level @dbclient now serves DebugModel and its instances
Performance Considerations
Encryption Performance
- XChaCha20-Poly1305 is ~2x faster than AES-256-GCM
- Key derivation is NOT cached by default for security
- Use request-level caching for bulk operations:
Familia::Encryption.with_request_cache do
# Bulk operations with cached key derivation
1000.times do |i|
User.create(email: "user#{i}@example.com")
end
end
Connection Pooling
- Use connection_provider for multi-threaded environments
- Pool size should match thread count
- Connections are thread-safe when using provider pattern
- Wrap scripts, jobs and tests in
Familia.with_connection_scopeto hold one connection per URI for the block and close it afterwards
Feature Performance Impact
- Encryption: ~10-20% overhead for field access
- Relationships: O(1) for indexed lookups
- Quantization: Minimal overhead, improves storage efficiency
- Safe Dump: Lazy evaluation, only computed when called
Migration Guide
From Familia v1.x to v2.0
- Replace
redis_uriwithuriin configuration - Update feature syntax from mixins to
featuredeclarations - Migrate from
global_prefix toclass_for class-level methods - Update encryption configuration to new provider system
Connection Provider Pattern
- Familia v2.0 uses redis-rb gem internally
- Connection providers must return Redis connections
- Uses RedisClient middleware architecture internally via redis-rb
Summary
Familia v2.0.0-pre series provides a comprehensive ORM for Valkey/Redis with:
- Modular Feature System: Isolated, configurable features per class
- Advanced Security: Field-level encryption with multiple providers
- Flexible Relationships: Automatic management with clean Ruby syntax
- Performance Optimized: Connection pooling, sampling, and caching
- Production Ready: Debug logging, monitoring, and thread safety
For additional documentation and examples, see the Familia GitHub repository.
class ActivityTracker < Familia::Horreum feature :relationships
identifier_field :activity_id field :activity_id, :user_id, :activity_type, :data, :created_at
# Class-level tracking with automatic management class_participates_in :user_activities, score: :created_at class_participates_in :activity_by_type, score: -> { "#activity_type:#created_at".hash } end
Create and save activity (automatic tracking)
activity = ActivityTracker.new( activity_id: 'act123', user_id: 'user456', activity_type: 'login', created_at: Familia.now.to_i ) activity.save # Automatically added to both tracking collections
Query recent activities (last hour)
hour_ago = (Familia.now - 1.hour).to_i recent_activities = ActivityTracker.user_activities.range_by_score( hour_ago, '+inf' )
Get activities by type in time range
login_hash_start = "login:#hour_ago".hash login_hash_end = "login:#Familia.now.to_i".hash login_activities = ActivityTracker.activity_by_type.range_by_score( login_hash_start, login_hash_end )
---
## Data Type Usage Patterns
### Advanced Sorted Set Operations
Leverage Valkey/Redis sorted sets for rankings, time series, and scored data.
```ruby
class Leaderboard < Familia::Horreum
identifier_field :game_id
field :game_id, :name
sorted_set :scores
end
leaderboard = Leaderboard.new(game_id: "game1", name: "Daily Challenge")
# Add player scores
leaderboard.scores.add(1500, "player1")
leaderboard.scores.add(2300, "player2")
leaderboard.scores.add(1800, "player3")
# Get top 10 players (highest scores first)
top_players = leaderboard.scores.revrange(0, 9, withscores: true)
# => [["player2", 2300.0], ["player3", 1800.0], ["player1", 1500.0]]
# Get player rank (0-indexed, lower scores = lower rank)
rank = leaderboard.scores.rank("player1") # => 0
rev_rank = leaderboard.scores.revrank("player1") # => 2 (highest to lowest)
# Get score range
mid_tier = leaderboard.scores.rangebyscore(1000, 2000, withscores: true)
# Increment score atomically
leaderboard.scores.increment("player1", 100) # Add 100 to existing score
List-Based Queues and Feeds
Use Valkey/Redis lists for queues, feeds, and ordered data.
class TaskQueue < Familia::Horreum
identifier_field :queue_name
field :queue_name
list :tasks
end
class ActivityFeed < Familia::Horreum
identifier_field :user_id
field :user_id
list :activities
end
# Task queue operations
queue = TaskQueue.new(queue_name: "email_processing")
# Add tasks to queue (right push)
queue.tasks.push({
type: "send_email",
recipient: "user@example.com",
template: "welcome"
}.to_json)
# Process tasks (left pop - FIFO)
next_task = queue.tasks.pop # Atomic pop from left
task_data = JSON.parse(next_task) if next_task
# Activity feed with size limit
feed = ActivityFeed.new(user_id: "user123")
# Add activity (keep last 100)
feed.activities.unshift("User logged in at #{Familia.now}")
feed.activities.trim(0, 99) # Keep only last 100 items
# Get recent activities
recent = feed.activities.range(0, 9) # Get 10 most recent
Hash-Based Configuration Storage
Store structured configuration and key-value data.
class UserPreferences < Familia::Horreum
identifier_field :user_id
field :user_id
hash :settings
hash :feature_flags
end
prefs = UserPreferences.new(user_id: "user123")
# UnsortedSet individual preferences
prefs.settings["theme"] = "dark"
prefs.settings["notifications"] = "true"
prefs.settings["timezone"] = "UTC-5"
# Batch set multiple values
prefs.feature_flags.update(
"beta_ui" => "true",
"new_dashboard" => "false",
"advanced_features" => "true"
)
# Get preferences
theme = prefs.settings["theme"] # => "dark"
all_settings = prefs.settings.to_h # => Hash of all settings
# Check feature flags
beta_enabled = prefs.feature_flags["beta_ui"] == "true"
Error Handling and Validation
Connection Error Handling
Robust error handling for Valkey/Redis connection issues.
class ResilientService < Familia::Horreum
field :name, :data
def self.with_fallback(&block)
retries = 3
begin
yield
rescue Redis::ConnectionError, Redis::TimeoutError => e
retries -= 1
if retries > 0
sleep(0.1 * (4 - retries)) # Exponential backoff
retry
else
Familia.warn "Database operation failed after retries: #{e.}"
nil # Return nil or handle gracefully
end
end
end
def save_with_fallback
self.class.with_fallback { save }
end
end
Data Validation Patterns
Implement validation in model classes.
class User < Familia::Horreum
field :email, :username, :age
def valid?
errors.clear
validate_email
validate_username
validate_age
errors.empty?
end
def errors
@errors ||= []
end
private
def validate_email
unless email&.include?('@')
errors << "Email must be valid"
end
end
def validate_username
if username.nil? || username.length < 3
errors << "Username must be at least 3 characters"
end
end
def validate_age
unless age.is_a?(Integer) && age > 0
errors << "Age must be a positive integer"
end
end
end
# Usage
user = User.new(email: "invalid", username: "ab", age: -5)
if user.valid?
user.save
else
puts "Validation errors: #{user.errors.join(', ')}"
end
Performance Optimization
Pipelined Bulk Loading
Load multiple objects efficiently with a single pipelined Redis batch.
# Before: N×2 commands (EXISTS + HGETALL per object)
users = ids.map { |id| User.find_by_id(id) }
# For 14 objects: 28 Redis commands
# After: 1 pipelined batch
users = User.load_multi(ids)
# For 14 objects: 1 batch with 14 HGETALL commands (2× faster)
# Load by full dbkeys
users = User.load_multi_by_keys(['user:123:object', 'user:456:object'])
# Filter out nils for missing objects
existing_users = User.load_multi(ids).compact
Optional EXISTS Check Optimization
Skip the EXISTS check for 50% reduction in Redis commands when keys are known to exist.
# Default behavior (2 commands: EXISTS + HGETALL)
user = User.find_by_id(123)
# Optimized (1 command: HGETALL only)
user = User.find_by_id(123, check_exists: false)
When to use check_exists: false:
- Loading from sorted set results (keys guaranteed to exist)
- High-throughput API endpoints
- Bulk operations with known-existing keys
Batch Operations
Minimize Valkey/Redis round trips with batch operations.
users = 100.times.map do |i|
User.new(name: "User #{i}", email: "user#{i}@example.com")
end
# Persist every record in one MULTI/EXEC. Each record's unique-index
# checks and claims run before the MULTI opens.
Familia.atomic_write(*users)
# Load many records in one pipeline of HGETALL commands
User.load_multi(users.map(&:identifier))
Do not wrap save in pipelined or transaction. It needs command
replies to check unique indexes and existence, so it raises
Familia::OperationModeError there, as do create!, build,
atomic_write and the partial writers commit_fields, save_fields,
multi_field_update and multi_field_fast_write. Without
Familia.atomic_write, save the records one at a time; each save runs
its own MULTI/EXEC. See
Transaction Safety, rules 1 and 4.
Index Rebuilding
Auto-generated rebuild methods for unique and multi indexes with zero downtime.
class User < Familia::Horreum
feature :relationships
unique_index :email, :email_lookup
end
# Rebuild class-level unique index
User.rebuild_email_lookup
# With progress tracking
User.rebuild_email_lookup(batch_size: 100) do |progress|
puts "#{progress[:completed]}/#{progress[:total]}"
end
# Instance-scoped index rebuild
company.rebuild_badge_index
When to use:
- After data migrations or bulk imports
- Recovering from index corruption
- Adding indexes to existing data
Each rebuild holds an advisory lock at <index_key>:rebuild-lock. A rebuild of
an index that is already being rebuilt raises
Familia::RebuildInProgressError rather than running concurrently. A rebuild
whose batch overran the lock TTL and lost the lock to another rebuild raises
Familia::RebuildLockLostError at the next batch or before the swap, discards
its own temporary key, and leaves the live index untouched. Calling a rebuild
inside an enclosing Familia.transaction or pipeline raises
Familia::OperationModeError, because the lock's SET NX would only be queued
there. The temporary key used for the swap expires after 300 seconds between
batches, and the swap fails closed: it is one Lua script that verifies the lock
token before the RENAME, and a rebuild that cannot complete the swap raises
rather than leaving a partial or stale index reported as current. Any batch
failure discards the whole temporary key. See
Concurrency and temporary keys
for the sweep used to clear temporary keys left by older versions, and the
precondition for running it.
The rebuild_<index> method of a multi_index checks every per-value bucket
key it will write before it deletes any bucket. If one of them holds a type
other than set, or is a set field or bucket of an existing record of the same
class with a longer identifier, it raises Familia::IndexBucketConflictError
without deleting or writing anything. A record exists when its hash key holds a
hash. The rebuild also raises when a key it would write or delete has more
candidate owner records than RecordKeyOwnership::MAX_CANDIDATE_SPLITS (8),
whose owner it does not check. The error's conflicts maps each key to the
type it holds, or to :too_many_candidate_owners, and its owners maps each
key of another record to that record's hash key. Inspect those keys: the
rebuild can run once none of them holds a type other than set, belongs to
another record or has too many candidate owners. repair_multi_indexes!
records such a rebuild in :skipped with reason: :bucket_conflict and goes
on with the next scope or index, and repair_all! then returns status
:partial_failure. See
Multi-value index rebuilds.
Memory Optimization
Efficient memory usage patterns.
class CacheEntry < Familia::Horreum
feature :expiration
field :key, :value, :created_at
default_expiration 1.hour
# Use shorter field names to reduce memory
field :k, :v, :c # Instead of key, value, created_at
# Compress large values
def value=(val)
@value = val.length > 1000 ? Zlib.deflate(val) : val
end
def value
val = @value || ""
val.start_with?("\x78\x9c") ? Zlib.inflate(val) : val
rescue Zlib::DataError
@value # Return original if decompression fails
end
end
Connection Pool Sizing
Configure connection pools based on application needs.
require 'connection_pool' # list connection_pool in your Gemfile
# High-throughput application
HIGH_THROUGHPUT_POOL = ConnectionPool::Wrapper.new(size: 25, timeout: 5) do
Redis.new(url: ENV['REDIS_URL'])
end
Familia.connection_provider = ->(_uri) { HIGH_THROUGHPUT_POOL }
# Memory-constrained environment
SMALL_POOL = ConnectionPool::Wrapper.new(size: 5, timeout: 10) do
Redis.new(url: ENV['REDIS_URL'])
end
Familia.connection_provider = ->(_uri) { SMALL_POOL }
Both build the pool once at boot. A single-pool provider like this ignores the
uri it is given, so it only works when every model shares one logical
database; see Connection Provider Pattern for
the per-URI form.
Migration and Upgrading
From v1.x to v2.0
Key changes and migration steps.
# OLD v1.x syntax
class User < Familia
identifier :email
string :name
list :sessions
end
# NEW v2.0 syntax
class User < Familia::Horreum
identifier_field :email # Updated method name
field :name # Generic field method
list :sessions # Data types unchanged
end
# Feature activation (NEW)
class User < Familia::Horreum
feature :expiration # Explicit feature activation
feature :safe_dump
identifier_field :email
field :name
list :sessions
default_expiration 24.hours # Feature-specific methods
safe_dump_field :name # Feature-specific methods
end
Encryption Migration
Migrating existing fields to encrypted storage.
# Step 1: Add feature without changing existing fields
class User < Familia::Horreum
feature :encrypted_fields # Add feature
field :name, :email
field :api_key # Still plaintext during migration
end
# Step 2: Migrate data with dual read/write
class User < Familia::Horreum
feature :encrypted_fields
field :name, :email
encrypted_field :api_key # Now encrypted
# Temporary migration method
def migrate_api_key!
if raw_api_key = dbclient.hget(dbkey, "api_key") # Read old plaintext
self.api_key = raw_api_key # Write as encrypted
dbclient.hdel(dbkey, "api_key") # Remove plaintext
save
end
end
end
# Step 3: Run migration for existing users
User.instances.each(&:migrate_api_key!)
Testing Patterns
Test Helpers and Utilities
Common patterns for testing Familia applications.
# test_helper.rb
require 'familia'
# Use separate Valkey/Redis database for tests
Familia.configure do |config|
config.uri = ENV.fetch('REDIS_TEST_URI', 'redis://localhost:2525/3')
end
module TestHelpers
def setup_redis
# Clear test database
Familia.dbclient.flushdb
end
def teardown_redis
Familia.dbclient.flushdb
end
def create_test_user(**attrs)
User.new({
email: "test@example.com",
name: "Test User",
created_at: Familia.now.to_i
}.merge(attrs))
end
end
# In test files
class UserTest < Minitest::Test
include TestHelpers
def setup
setup_redis
end
def teardown
teardown_redis
end
def test_user_creation_with_automatic_indexing
user = create_test_user(name: "Alice")
user.save # Automatically adds to class-level indexes
assert user.exists?
assert_equal "Alice", user.name
# Test automatic indexing (if using unique_index)
if User.respond_to?(:find_by_email)
found_user = User.find_by_email(user.email)
assert_equal user.identifier, found_user.identifier
end
end
def test_relationships_with_clean_syntax
user = create_test_user
user.save # Automatic class-level tracking
domain = Domain.new(domain_id: "test_domain", name: "test.com")
domain.save # Automatic class-level tracking
# Test clean relationship syntax
user.domains << domain # Ruby-like collection syntax
assert domain.in_user_domains?(user.identifier)
assert user.domains.member?(domain.identifier)
end
def test_encrypted_fields_concealment
setup_encryption_keys
vault = SecureVault.new(
name: "Test Vault",
secret_key: "super-secret-123",
api_token: "sk-1234567890"
)
vault.save
# Test ConcealedString behavior
assert_instance_of Familia::Features::EncryptedFields::ConcealedString, vault.secret_key
assert_equal "[CONCEALED]", vault.secret_key.to_s
assert_equal "super-secret-123", vault.secret_key.reveal
# Test JSON safety
json_data = vault.to_json
refute_includes json_data, "super-secret-123"
assert_includes json_data, "[CONCEALED]"
end
def test_transient_fields_non_persistence
form = LoginForm.new(
username: "testuser",
password: "secret123",
csrf_token: "abc123"
)
form.save
# Reload and verify transient fields not persisted
reloaded = LoginForm.load(form.identifier)
assert_equal "testuser", reloaded.username
assert_nil reloaded.password
assert_nil reloaded.csrf_token
end
def test_object_identifier_generation
# Test UUID generation
doc = UuidDocument.create(title: "Test Doc")
assert_match(/\A[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}\z/i, doc.objid)
# Test hex generation
session = HexSession.create(user_id: "123")
assert_match(/\A[0-9a-f]+\z/i, session.objid)
assert_equal 16, session.objid.length
end
def test_quantization_time_bucketing
= Time.parse("2024-12-15 14:37:23")
bucket = MetricsBucket.()
assert_equal "20241215_1430", bucket # 10-minute bucket
# Test multiple timestamps in same bucket
= Time.parse("2024-12-15 14:39:45")
bucket2 = MetricsBucket.()
assert_equal bucket, bucket2
end
def test_external_identifier_mapping
user = ExternalUser.new(name: "External User")
user.save
# Test external ID is derived from objid
assert_not_nil user.objid
assert_not_nil user.extid
assert user.extid.start_with?("ext_")
# Test deterministic generation
user2 = ExternalUser.new(objid: user.objid, name: "External User")
assert_equal user.extid, user2.extid
end
private
def setup_encryption_keys
test_keys = {
v1: ['a' * 32].pack('m0'), # strict Base64, as encryption keys require
v2: ['b' * 32].pack('m0')
}
Familia.configure do |config|
config.encryption_keys = test_keys
config.current_key_version = :v1
config.encryption_personalization = 'TestApp-Test'
end
end
end
Resources and References
Key Configuration
Essential configuration options for Familia v2.0.0-pre.
Familia.configure do |config|
# Basic Valkey/Redis connection
config.uri = ENV['REDIS_URL'] || 'redis://localhost:6379/0'
# Connection provider for pooling (optional)
config.connection_provider = MyConnectionProvider
# Encryption configuration (for encrypted_fields feature)
config.encryption_keys = {
v1: ENV['FAMILIA_ENCRYPTION_KEY_V1'],
v2: ENV['FAMILIA_ENCRYPTION_KEY_V2']
}
config.current_key_version = :v2
# Debugging and logging
config.debug = ENV['FAMILIA_DEBUG'] == 'true'
config.enable_database_logging = ENV['FAMILIA_LOG_REDIS'] == 'true'
end
Documentation Links
- Familia Repository
- Wiki Home
- Feature System Guide
- Relationships Guide
- Encrypted Fields Overview
- Connection Pooling Guide
Version Information
- Current Version: v2.0.0
- Ruby Compatibility: 3.2+
- Redis Compatibility: 6.0+ (Valkey compatible)
This technical reference covers the major components and usage patterns available in Familia v2.0. For complete API documentation, see the generated YARD docs and wiki guides.