Module: Familia::Features::Relationships::Indexing::MultiIndexGenerators
- Defined in:
- lib/familia/features/relationships/indexing/multi_index_generators.rb
Overview
Generators for multi-value index (1:many) methods
Multi-value indexes use UnsortedSet DataType for grouping objects by field value. Each field value gets its own set of object identifiers.
Example: multi_index :department, :dept_index, within: Company
Generates on Company (destination):
- company.sample_from_department(dept, count=1)
- company.find_all_by_department(dept)
- company.dept_index_for(dept_value)
- company.rebuild_dept_index
Generates on Employee (self):
- employee.add_to_company_dept_index(company)
- employee.remove_from_company_dept_index(company)
- employee.update_in_company_dept_index(company, old_dept)
Constant Summary collapse
- MAX_FIELD_VALUE_LENGTH =
Maximum recommended length for field values used in index keys. Longer values are allowed but will trigger a warning.
256- BUCKET_KEY_TYPE =
Redis type of every per-value bucket key. The factories below build each bucket as a Familia::UnsortedSet.
'set'- WRITABLE_BUCKET_KEY_TYPES =
Types a key may hold before the rebuild writes it as a bucket: a set, or "none" when the key does not exist.
[BUCKET_KEY_TYPE, 'none'].freeze
Class Method Summary collapse
-
.blank_field_value?(field_value) ⇒ Boolean
Whether +field_value+ is blank: nil, or empty or whitespace only as a String.
-
.bucket_key_prefix(owner, index_name) ⇒ String
Returns the literal key prefix of the per-value buckets of +index_name+ on +owner+, for example "company:c-1:dept_index:" for a scope instance or "customer:role_index:" for a class.
-
.check_live_bucket_keys!(client, owner, bucket_keys, batch_size: 100) ⇒ nil
Raises Familia::IndexBucketConflictError when one of +bucket_keys+ holds a type other than a set, is another existing record's key (see RecordKeyOwnership.other_record_key_owner) or has more candidate owners than the rebuild checks.
-
.clear_index_buckets(client, owner, index_name, live_bucket_keys:, batch_size: 100) {|key, cleared| ... } ⇒ Integer
Deletes the per-value bucket sets of +index_name+ on +owner+, including buckets for field values that no object holds any more.
-
.delete_index_buckets(client, bucket_keys, batch_size: 100) {|key, cleared| ... } ⇒ Integer
Deletes +bucket_keys+ for .clear_index_buckets, one DEL per key in one pipeline per batch, and counts only the keys DEL removed.
-
.each_index_bucket_slice(client, owner, index_name, batch_size: 100) {|buckets, others| ... } ⇒ Enumerator?
Yields the keys of the per-value bucket sets of +index_name+ on +owner+, in slices of up to +batch_size+ keys.
-
.generate_factory_method(indexed_class, scope_class, index_name) ⇒ Object
Generates the factory method ON THE SCOPE CLASS (Company when within: Company): - company.index_name_for(field_value) - DataType factory (always needed).
-
.generate_factory_method_class(indexed_class, index_name) ⇒ Object
Generates class-level factory method: - Customer.role_index_for(field_value) -> UnsortedSet.
-
.generate_mutation_methods_class(indexed_class, field, index_name) ⇒ Object
Generates instance mutation methods for class-level indexes: - customer.add_to_class_role_index - customer.remove_from_class_role_index - customer.update_in_class_role_index(old_value).
-
.generate_mutation_methods_self(indexed_class, field, scope_class, index_name) ⇒ Object
Generates mutation methods ON THE INDEXED CLASS (Employee): - employee.add_to_company_dept_index(company) - employee.remove_from_company_dept_index(company) - employee.update_in_company_dept_index(company, old_dept).
-
.generate_query_methods_class(indexed_class, field, index_name) ⇒ Object
Generates class-level query methods: - Customer.find_all_by_role(value) -> [Customer, ...] - Customer.sample_from_role(value, count) -> random sample - Customer.rebuild_role_index -> rebuild index.
-
.generate_query_methods_destination(indexed_class, field, scope_class, index_name) ⇒ Object
Generates query methods ON THE SCOPE CLASS (Company when within: Company): - company.sample_from_department(dept, count=1) - random sampling - company.find_all_by_department(dept) - all objects - company.rebuild_dept_index - rebuild index.
-
.record_live_bucket_conflict(conflicts, owners, key, type, record_key) ⇒ void
Adds +key+ to +conflicts+ for .check_live_bucket_keys! when it has more candidate owners than RecordKeyOwnership checks, is another record's key, or holds a type other than a set.
-
.select_index_buckets(keys, bucket_prefix) ⇒ Array<String>
Returns the +keys+ that start with the literal +bucket_prefix+.
-
.setup(indexed_class:, field:, index_name:, within:, query:) ⇒ Object
Main setup method that orchestrates multi-value index creation.
-
.validate_field_value(field_value, context: 'index') ⇒ String?
Validates a field value for use in index key construction.
Class Method Details
.blank_field_value?(field_value) ⇒ Boolean
Whether +field_value+ is blank: nil, or empty or whitespace only as a String. A multi_index does not index a blank value, and the generated methods of an owner do not read or write the key the factories name for one on that owner (see .bucket_key_prefix).
144 145 146 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 144 def blank_field_value?(field_value) field_value.nil? || Familia.blank_text?(field_value) end |
.bucket_key_prefix(owner, index_name) ⇒ String
Returns the literal key prefix of the per-value buckets of +index_name+ on +owner+, for example "company:c-1:dept_index:" for a scope instance or "customer:role_index:" for a class.
The factories pass a field value through .validate_field_value and name its bucket Familia.join(index_name, validated) under +owner+'s key. This method joins +index_name+ with '' instead, which leaves the index name followed by the delimiter. Every bucket of a non-blank field value starts with that prefix.
A blank value (nil, empty or whitespace only) is the exception.
Validation turns it into nil, which Familia.join drops, so the
factories name its key "set :tags beside
multi_index :tag, :tags, within: Scope, or class_set :roles
beside multi_index :role, :roles. So the multi_index methods
of +owner+ do not read or write it: add_to_, update_in_,
remove_from_* and the rebuild skip blank values, find_all_by_*
and sample_from_* return [] for them, IndexDescriptor#each_record
raises for them, and the rebuild SCAN of +owner+ does not match
the key (see .blank_field_value?).
The key can still be a bucket of another multi_index. An identifier that contains the delimiter can make it a bucket of an instance-scoped multi_index within the scope class in a shorter scope: with scopes "c-1" and "c-1:dept_index:x", the key of a blank value in "c-1:dept_index:x" is the bucket of "x:dept_index" in "c-1". An identifier that is the name of a class-level multi_index of the scope class, or starts with that name and the delimiter, makes it a bucket of that index, with or without a delimiter in the identifier: the key of a blank value in scope "region_index" is the class-level "region_index" bucket of the value +index_name+. audit_multi_indexes of that other index reads it as that bucket, and the rebuild of that index deletes or rewrites it. This does not happen when the scope class declares a set field named +index_name+ and +owner+ exists, because the key is then a key of +owner+ (see RecordKeyOwnership.other_record_key).
133 134 135 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 133 def bucket_key_prefix(owner, index_name) Familia::UnsortedSet.new(Familia.join(index_name, ''), parent: owner).dbkey end |
.check_live_bucket_keys!(client, owner, bucket_keys, batch_size: 100) ⇒ nil
Raises Familia::IndexBucketConflictError when one of +bucket_keys+ holds a type other than a set, is another existing record's key (see RecordKeyOwnership.other_record_key_owner) or has more candidate owners than the rebuild checks.
After the clearing phase, the rebuild adds every object to the bucket of its field value. A key of another type there is not a bucket and may be another record's data, such as the list of a record named after a class-level index when a live field value equals that list's name. SADD into it raises WRONGTYPE, which would stop the rebuild after clearing had already emptied the other buckets, and deleting it could destroy that data. A set that is another record's key would receive index members. So the rebuild checks every key it will write before it deletes any.
308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 308 def check_live_bucket_keys!(client, owner, bucket_keys, batch_size: 100) conflicts = {} owners = {} names = RecordKeyOwnership.record_set_key_names(owner) bucket_keys.each_slice(batch_size) do |keys| types = client.pipelined do |pipe| keys.each { |key| pipe.type(key) } end record_keys = RecordKeyOwnership.other_record_key_owners(keys, owner, names: names) keys.each_with_index do |key, idx| record_live_bucket_conflict(conflicts, owners, key, types[idx], record_keys[idx]) end end raise Familia::IndexBucketConflictError.new(conflicts, owners) unless conflicts.empty? nil end |
.clear_index_buckets(client, owner, index_name, live_bucket_keys:, batch_size: 100) {|key, cleared| ... } ⇒ Integer
Deletes the per-value bucket sets of +index_name+ on +owner+, including buckets for field values that no object holds any more.
+live_bucket_keys+ are the keys the rebuild writes after this returns. They are checked with .check_live_bucket_keys! first. Then every bucket under the prefix is found before any is deleted, and a key whose owner was not checked (see .each_index_bucket_slice) raises as well. So a key the rebuild could not write, or could not tell from another record's key, stops it before any key is deleted.
Only the bucket keys .each_index_bucket_slice yields are deleted. A key under the prefix that holds another type is left alone, and so is a set that is another existing record's key. Each key gets its own DEL in one pipeline per batch, so a key that was already gone when its DEL ran is neither yielded nor counted.
244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 244 def clear_index_buckets(client, owner, index_name, live_bucket_keys:, batch_size: 100, &on_delete) check_live_bucket_keys!(client, owner, live_bucket_keys, batch_size: batch_size) stale = [] unchecked = {} each_index_bucket_slice(client, owner, index_name, batch_size: batch_size) do |buckets, others| stale.concat(buckets) others.each do |key, record_key| unchecked[key] = record_key if record_key == RecordKeyOwnership::TOO_MANY_CANDIDATES end end raise Familia::IndexBucketConflictError, unchecked unless unchecked.empty? delete_index_buckets(client, stale, batch_size: batch_size, &on_delete) end |
.delete_index_buckets(client, bucket_keys, batch_size: 100) {|key, cleared| ... } ⇒ Integer
Deletes +bucket_keys+ for .clear_index_buckets, one DEL per key in one pipeline per batch, and counts only the keys DEL removed.
269 270 271 272 273 274 275 276 277 278 279 280 281 282 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 269 def delete_index_buckets(client, bucket_keys, batch_size: 100, &on_delete) cleared = 0 bucket_keys.each_slice(batch_size) do |buckets| deleted = client.pipelined { |pipe| buckets.each { |key| pipe.del(key) } } buckets.zip(deleted) do |key, count| next unless count.to_i.positive? cleared += 1 on_delete&.call(key, cleared) end end cleared end |
.each_index_bucket_slice(client, owner, index_name, batch_size: 100) {|buckets, others| ... } ⇒ Enumerator?
Yields the keys of the per-value bucket sets of +index_name+ on +owner+, in slices of up to +batch_size+ keys.
The SCAN pattern is "TYPE subcommand." under 6.0.0, and README.md lists the
prerequisite "Valkey/Redis*: 6.0+". Each returned key must
also start with the literal prefix (see .select_index_buckets).
A key can start with the literal prefix and still be another record's key, because an identifier may contain the delimiter. "company:c-1:dept_index:x:dept_index:ops" is both the bucket of value "x:dept_index:ops" in scope "c-1" and the "ops" bucket of scope "c-1:dept_index:x". Keys that RecordKeyOwnership.other_record_key assigns to an existing record with a longer identifier are yielded in +others+, not in +buckets+, so the rebuild does not delete them and the class-level audit does not read them. So is a key with more candidate owners than RecordKeyOwnership::MAX_CANDIDATE_SPLITS, whose owner is not checked. The candidates of the keys in a slice are checked in pipelines of at most RecordKeyOwnership::MAX_CANDIDATE_BATCH_BYTES of identifiers.
The key "
193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 193 def each_index_bucket_slice(client, owner, index_name, batch_size: 100) return enum_for(__method__, client, owner, index_name, batch_size: batch_size) unless block_given? bucket_prefix = bucket_key_prefix(owner, index_name) names = RecordKeyOwnership.record_set_key_names(owner) pattern = "#{Familia.escape_glob(bucket_prefix)}*" client.scan_each(match: pattern, count: batch_size, type: BUCKET_KEY_TYPE) .each_slice(batch_size) do |keys| candidates = select_index_buckets(keys, bucket_prefix) next if candidates.empty? owners = RecordKeyOwnership.other_record_key_owners(candidates, owner, names: names) buckets = [] others = {} candidates.zip(owners) do |key, record_key| record_key ? others[key] = record_key : buckets << key end yield buckets, others end nil end |
.generate_factory_method(indexed_class, scope_class, index_name) ⇒ Object
Generates the factory method ON THE SCOPE CLASS (Company when within: Company):
- company.index_name_for(field_value) - DataType factory (always needed)
This method is required by mutation methods even when query: false
414 415 416 417 418 419 420 421 422 423 424 425 426 427 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 414 def generate_factory_method(indexed_class, scope_class, index_name) actual_scope_class = Familia.resolve_class(scope_class) idx_name = index_name actual_scope_class.class_eval do # Helper method to get index set for a specific field value # This acts as a factory for field-value-specific DataTypes define_method(:"#{index_name}_for") do |field_value| validated = MultiIndexGenerators.validate_field_value(field_value, context: "#{self.class.name}.#{idx_name}") index_key = Familia.join(index_name, validated) Familia::UnsortedSet.new(index_key, parent: self, class: indexed_class, reference: true) end end end |
.generate_factory_method_class(indexed_class, index_name) ⇒ Object
Generates class-level factory method:
- Customer.role_index_for(field_value) -> UnsortedSet
The factory validates field values for data quality. Glob pattern characters (*, ?, [, ]) in field values are allowed but trigger warnings since they can be confusing during debugging.
787 788 789 790 791 792 793 794 795 796 797 798 799 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 787 def generate_factory_method_class(indexed_class, index_name) # Capture index_name for use in validation context idx_name = index_name indexed_class.define_singleton_method(:"#{index_name}_for") do |field_value| # Validate field value and use the validated string for consistent key format. # Validation returns nil for nil/empty values, string otherwise. # We allow nil through (creates a "null" index key) but use the validated # string to ensure consistent type handling in key construction. validated = MultiIndexGenerators.validate_field_value(field_value, context: "#{name}.#{idx_name}") index_key = Familia.join(index_name, validated) Familia::UnsortedSet.new(index_key, parent: self, class: indexed_class, reference: true) end end |
.generate_mutation_methods_class(indexed_class, field, index_name) ⇒ Object
Generates instance mutation methods for class-level indexes:
- customer.add_to_class_role_index
- customer.remove_from_class_role_index
- customer.update_in_class_role_index(old_value)
These are auto-called on save/destroy when auto-indexing is enabled.
949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 949 def generate_mutation_methods_class(indexed_class, field, index_name) indexed_class.class_eval do method_name = :"add_to_class_#{index_name}" Familia.debug("[MultiIndexGenerators] #{name} class method #{method_name}") define_method(method_name) do field_value = send(field) return unless field_value && !Familia.blank_text?(field_value) _ensure_persisted_before_index_write!(index_name) index_set = self.class.send("#{index_name}_for", field_value) index_set.add(identifier) end method_name = :"remove_from_class_#{index_name}" Familia.debug("[MultiIndexGenerators] #{name} class method #{method_name}") define_method(method_name) do field_value = send(field) return unless field_value && !Familia.blank_text?(field_value) index_set = self.class.send("#{index_name}_for", field_value) index_set.remove(identifier) end method_name = :"update_in_class_#{index_name}" Familia.debug("[MultiIndexGenerators] #{name} class method #{method_name}") define_method(method_name) do |old_field_value| return unless old_field_value new_field_value = send(field) return if old_field_value == new_field_value _ensure_persisted_before_index_write!(index_name) # Get the index sets for old and new values old_set = self.class.send("#{index_name}_for", old_field_value) # Use DataType's serialize_value for consistency with add/remove methods. # This ensures the same serialization path is used across all index operations. serialized_id = old_set.serialize_value(identifier) # Use transaction for atomic remove + add to prevent data inconsistency transaction do |conn| # Remove from old index. A blank old value has no bucket: # its key may be a class-level related field of the same # name (see MultiIndexGenerators.bucket_key_prefix). unless MultiIndexGenerators.blank_field_value?(old_field_value) conn.srem(old_set.dbkey, serialized_id) end # Add to new index if present if new_field_value && !Familia.blank_text?(new_field_value) new_set = self.class.send("#{index_name}_for", new_field_value) conn.sadd(new_set.dbkey, serialized_id) end end end end end |
.generate_mutation_methods_self(indexed_class, field, scope_class, index_name) ⇒ Object
Generates mutation methods ON THE INDEXED CLASS (Employee):
- employee.add_to_company_dept_index(company)
- employee.remove_from_company_dept_index(company)
- employee.update_in_company_dept_index(company, old_dept)
640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 640 def generate_mutation_methods_self(indexed_class, field, scope_class, index_name) scope_class_config = scope_class.config_name indexed_class.class_eval do method_name = :"add_to_#{scope_class_config}_#{index_name}" Familia.debug("[MultiIndexGenerators] #{name} method #{method_name}") define_method(method_name) do |scope_instance| return unless scope_instance # Before any write: an untrackable scope produces an index # entry that destroy! could never find again. _ensure_trackable_index_scope!(scope_instance) field_value = send(field) return unless field_value _ensure_persisted_before_index_write!(index_name, scope_instance) # A blank value is not indexed, as in the class-level index: # its key "<scope key>:<index>" may be a related field of the # scope (see MultiIndexGenerators.bucket_key_prefix). Its # tracker entry below is still recorded, so save keeps # refreshing this membership once the value is no longer # blank. unless MultiIndexGenerators.blank_field_value?(field_value) index_set = scope_instance.send("#{index_name}_for", field_value) index_set.add(identifier) end # Per-value entry: add_to_* is add-only, so the object may now # occupy several buckets and each needs its own cleanup entry. _record_index_scope(scope_class_config, index_name, scope_instance, field_value, cardinality: :multi) end method_name = :"remove_from_#{scope_class_config}_#{index_name}" Familia.debug("[MultiIndexGenerators] #{name} method #{method_name}") # @param scope_instance [Object] the scope holding the index # @param field_value [Object, nil] the value bucket to remove # from. Defaults to the object's current field value, # preserving the public single-argument call. destroy! cleanup # passes the value recorded at index time instead, since the # current one may have changed (or be nil on an # identifier-only instance). define_method(method_name) do |scope_instance, field_value = nil| return unless scope_instance field_value = send(field) if field_value.nil? return unless field_value # A blank value has no bucket (see add_to_*), so only its # tracker entry is dropped. unless MultiIndexGenerators.blank_field_value?(field_value) index_set = scope_instance.send("#{index_name}_for", field_value) index_set.remove(identifier) end # field_value is part of the multi entry key -- the bucket # just left is the entry to drop, not the membership as a # whole (other buckets may still hold this object). _unrecord_index_scope(scope_class_config, index_name, scope_instance, field_value, cardinality: :multi) end method_name = :"update_in_#{scope_class_config}_#{index_name}" Familia.debug("[MultiIndexGenerators] #{name} method #{method_name}") define_method(method_name) do |scope_instance, old_field_value = nil| return unless scope_instance _ensure_trackable_index_scope!(scope_instance) new_field_value = send(field) # A blank value is not indexed, as in the class-level index, # but the tracker below still records it (see add_to_*). # Neither a blank old value nor a blank new value touches # the key "<scope key>:<index>", which may be a related field # of the scope (see MultiIndexGenerators.bucket_key_prefix). index_value = new_field_value unless MultiIndexGenerators.blank_field_value?(new_field_value) retract_value = old_field_value unless MultiIndexGenerators.blank_field_value?(old_field_value) _ensure_persisted_before_index_write!(index_name, scope_instance) # Use Familia's transaction method for atomicity with DataType abstraction scope_instance.transaction do |_tx| # Remove from old index if provided - use helper method if retract_value old_index_set = scope_instance.send("#{index_name}_for", retract_value) old_index_set.remove(identifier) end # Add to new index if present - use helper method if index_value new_index_set = scope_instance.send("#{index_name}_for", index_value) new_index_set.add(identifier) end end # Mirror the bucket moves above into the tracker: drop the # entry for the retracted bucket, add one for the new bucket. # # Synced outside the block above, which behaves differently # depending on the caller. Standalone, scope_instance.transaction # opens a MULTI on the SCOPE's connection -- a tracker write # inside it would go to the wrong connection, since the tracker # belongs to this object. The tradeoff is that standalone, the # tracker sync is NOT atomic with the index write. During save # the block is reentrant (it joins the caller's MULTI via # Fiber[:familia_transaction]) so both land in that one # transaction and the tradeoff does not apply. _sync_multi_index_scope(scope_class_config, index_name, scope_instance, old_field_value, new_field_value) end end end |
.generate_query_methods_class(indexed_class, field, index_name) ⇒ Object
Generates class-level query methods:
- Customer.find_all_by_role(value) -> [Customer, ...]
- Customer.sample_from_role(value, count) -> random sample
- Customer.rebuild_role_index -> rebuild index
809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 809 def generate_query_methods_class(indexed_class, field, index_name) # find_all_by_role(value) # Uses load_multi for efficient batch loading (avoids N+1 queries) indexed_class.define_singleton_method(:"find_all_by_#{field}") do |field_value| # A blank value is not indexed (see MultiIndexGenerators.bucket_key_prefix) return [] if MultiIndexGenerators.blank_field_value?(field_value) Familia.assert_replies_available!("#{self}.find_all_by_#{field}") index_set = send("#{index_name}_for", field_value) identifiers = index_set.members load_multi(identifiers).compact end # sample_from_role(value, count) # Uses load_multi for efficient batch loading (avoids N+1 queries) indexed_class.define_singleton_method(:"sample_from_#{field}") do |field_value, count = 1| return [] if MultiIndexGenerators.blank_field_value?(field_value) Familia.assert_replies_available!("#{self}.sample_from_#{field}") index_set = send("#{index_name}_for", field_value) identifiers = index_set.sample(count) load_multi(identifiers).compact end # rebuild_role_index(batch_size:, &progress) # # For class-level indexes, we iterate all instances of the class. # The phases match the instance-scoped rebuild: load every object # and its field value, check every bucket key the rebuild will # write, delete the class's bucket sets under the escaped bucket # prefix (e.g. SCAN "customer:role_index:*"), then add each # object with a non-blank field value to the bucket of that # value. The key "customer:role_index" that the factory names for # a blank value is not matched or touched: it may be a # class-level related field of the same name. A set under # the prefix that is a set field of an existing record whose # identifier starts with the index name is left alone (see # RecordKeyOwnership.other_record_key). # # @param batch_size [Integer] Number of identifiers to process per batch # @yield [progress] Optional block called with progress updates; the # phases are :discovering, :loading, :clearing and :rebuilding # @return [Integer] Number of objects processed, or 0 when the class # has no instances collection # @raise [Familia::IndexBucketConflictError] if a bucket key the rebuild # will write holds a type other than set or is another existing record's # key, or if a key it will write or delete has more candidate owner # records than RecordKeyOwnership::MAX_CANDIDATE_SPLITS. The rebuild # raises before it deletes or writes anything. The error's #conflicts # maps each such key to its type or to # RecordKeyOwnership::TOO_MANY_CANDIDATES, and #owners maps each key of # another record to that record's hash key. # @note Like the instance-scoped rebuild, this caches every object, and # the clearing phase holds every bucket key under the prefix in memory # before it deletes any, so memory grows with the number of objects # and of bucket keys. indexed_class.define_singleton_method(:"rebuild_#{index_name}") do |batch_size: 100, &progress_block| Familia.assert_replies_available!("#{self}.rebuild_#{index_name}") # PHASE 1: Discover all field values and collect objects progress_block&.call(phase: :discovering, current: 0, total: 0) # Use class-level instances collection if available unless respond_to?(:instances) && instances.respond_to?(:members) Familia.warn "[Rebuild] Cannot rebuild class-level multi-index #{index_name}: " \ "no instances collection found. " \ "Ensure #{name} has class_sorted_set :instances or similar." return 0 # Return 0 for consistency - always return integer count end field_values = Set.new cached_objects = [] processed = 0 total_count = instances.size progress_block&.call(phase: :loading, current: 0, total: total_count) instances.members.each_slice(batch_size) do |identifiers| objects = load_multi(identifiers).compact cached_objects.concat(objects) objects.each do |obj| value = obj.send(field) field_values << value.to_s if value && !Familia.blank_text?(value) end processed += identifiers.size progress_block&.call(phase: :loading, current: processed, total: total_count) end # PHASE 2: Clear existing index sets (e.g. "customer:role_index:*") using SCAN. # The class prefix and delimiter are escaped like any other literal part. # The keys PHASE 3 writes, one per field value, are checked first. progress_block&.call(phase: :clearing, current: 0, total: field_values.size) live_bucket_keys = field_values.map { |value| send("#{index_name}_for", value).dbkey } MultiIndexGenerators.clear_index_buckets( dbclient, self, index_name, live_bucket_keys: live_bucket_keys, batch_size: batch_size ) do |key, cleared| progress_block&.call(phase: :clearing, current: cleared, total: field_values.size, key: key) end # PHASE 3: Rebuild from cached objects progress_block&.call(phase: :rebuilding, current: 0, total: cached_objects.size) processed = 0 cached_objects.each_slice(batch_size) do |objects| dbclient.multi do |conn| objects.each do |obj| field_value = obj.send(field) next unless field_value && !Familia.blank_text?(field_value) index_set = send("#{index_name}_for", field_value) conn.sadd(index_set.dbkey, index_set.serialize_value(obj.identifier)) end end processed += objects.size progress_block&.call(phase: :rebuilding, current: processed, total: cached_objects.size) end Familia.info "[Rebuild] Class-level multi-index #{index_name} rebuilt: " \ "#{field_values.size} field values, #{processed} objects" processed end end |
.generate_query_methods_destination(indexed_class, field, scope_class, index_name) ⇒ Object
Generates query methods ON THE SCOPE CLASS (Company when within: Company):
- company.sample_from_department(dept, count=1) - random sampling
- company.find_all_by_department(dept) - all objects
- company.rebuild_dept_index - rebuild index
438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 438 def generate_query_methods_destination(indexed_class, field, scope_class, index_name) # Resolve scope class using Familia pattern actual_scope_class = Familia.resolve_class(scope_class) # Get scope_class_config for method naming (needed for rebuild methods) scope_class_config = actual_scope_class.config_name # Generate instance sampling method (e.g., company.sample_from_department) actual_scope_class.class_eval do define_method(:"sample_from_#{field}") do |field_value, count = 1| # A blank value is not indexed (see MultiIndexGenerators.bucket_key_prefix) return [] if MultiIndexGenerators.blank_field_value?(field_value) Familia.assert_replies_available!("#{self.class}#sample_from_#{field}") index_set = send("#{index_name}_for", field_value) # i.e. UnsortedSet # Get random members efficiently (O(1) via SRANDMEMBER with count) # Returns array even for count=1 for consistent API index_set.sample(count).map do |id| indexed_class.find_by_identifier(id) end end # Generate bulk query method (e.g., company.find_all_by_department) define_method(:"find_all_by_#{field}") do |field_value| # A blank value is not indexed (see MultiIndexGenerators.bucket_key_prefix) return [] if MultiIndexGenerators.blank_field_value?(field_value) Familia.assert_replies_available!("#{self.class}#find_all_by_#{field}") index_set = send("#{index_name}_for", field_value) # i.e. UnsortedSet # Get all members from set index_set.members.map { |id| indexed_class.find_by_identifier(id) } end # Generate method to rebuild the multi-value index for this parent instance # # Multi-indexes create separate sets for each field value, so the rebuild runs in three phases: # 1. Loading: Load every object in the participation collection once and cache it, # collecting its field value. # 2. Clearing: Check every bucket key phase 3 will write (see @raise), then delete # this scope instance's bucket sets, including those of field values no object # holds any more. A SCAN over the escaped bucket prefix finds them, and only keys # that start with that literal prefix and hold a set are deleted. A set that # also belongs to an existing scope record with a longer identifier is left # alone (see RecordKeyOwnership.other_record_key). The SCAN pattern is the # escaped prefix followed by "*", e.g. "company:c-1:dept_index:*", so the key # "company:c-1:dept_index" that the factory names for a blank value is not # touched: it may be a related field of the same name on this scope class. # 3. Rebuilding: Add each cached object to the bucket of its field value # (no reload needed). An object with a blank field value is not indexed. # # @param batch_size [Integer] Number of identifiers to process per batch # @yield [progress] Optional block called with progress updates # @yieldparam progress [Hash] Progress information with keys: # - :phase [Symbol] Current phase (:loading, :clearing, :rebuilding) # - :current [Integer] Current item count # - :total [Integer] Total items (when known) # - :key [String] Bucket key just deleted (:clearing phase only) # @return [Integer, nil] Number of objects processed, or nil when the indexed # class has no participation relationship to this scope class # @raise [Familia::IndexBucketConflictError] if a bucket key the rebuild will # write holds a type other than set or is another existing record's key, or if # a key it will write or delete has more candidate owner records than # RecordKeyOwnership::MAX_CANDIDATE_SPLITS. The rebuild raises before it # deletes or writes anything. The error's #conflicts maps each such key to its # type or to RecordKeyOwnership::TOO_MANY_CANDIDATES, and #owners maps each # key of another record to that record's hash key. # # @example Basic rebuild # company.rebuild_dept_index # # @example With progress monitoring # company.rebuild_dept_index do |progress| # puts "#{progress[:phase]}: #{progress[:current]}/#{progress[:total]}" # end # # @example Memory-conscious rebuild for large collections # # Process in smaller batches to reduce memory footprint # company.rebuild_dept_index(batch_size: 50) # # @note Memory Considerations: # This method caches all objects in memory during rebuild to avoid duplicate # database loads. For very large collections (>100k objects), monitor memory usage # and consider processing in chunks or using a streaming approach if memory # constraints are encountered. The batch_size parameter controls Redis I/O # batching but does not affect memory usage since all objects are cached. # The clearing phase also holds every bucket key under the prefix in memory # before it deletes any (see MultiIndexGenerators.clear_index_buckets), so # memory grows with the number of bucket keys as well. # define_method(:"rebuild_#{index_name}") do |batch_size: 100, &progress_block| Familia.assert_replies_available!("#{self.class}#rebuild_#{index_name}") # PHASE 1: Find the collection containing the indexed objects # Look for a participation relationship where indexed_class participates in this scope_class collection_name = nil # Check if indexed_class has participation to this scope_class if indexed_class.respond_to?(:participation_relationships) participation = indexed_class.participation_relationships.find do |rel| rel.target_class == self.class end collection_name = participation&.collection_name if participation end # Get the collection DataType if we found a participation relationship collection = collection_name ? send(collection_name) : nil if collection # PHASE 2: Load objects once and cache them for both discovery and rebuilding # This avoids duplicate load_multi calls (previous approach loaded twice) progress_block&.call(phase: :loading, current: 0, total: collection.size) field_values = Set.new cached_objects = [] processed = 0 collection.members.each_slice(batch_size) do |identifiers| # Load objects in batches - SINGLE LOAD for both phases objects = indexed_class.load_multi(identifiers).compact cached_objects.concat(objects) objects.each do |obj| value = obj.send(field) # Only track non-nil, non-empty field values field_values << value.to_s if value && !Familia.blank_text?(value) end processed += identifiers.size progress_block&.call(phase: :loading, current: processed, total: collection.size) end # PHASE 3: Clear this scope instance's field-value-specific index sets. # SCAN also finds orphaned sets left by field values no object holds any more. # The identifier in the prefix (e.g. "company:c-*:dept_index:") is escaped, # so a glob character in it cannot widen the match past that literal prefix. # The keys PHASE 4 writes are checked first: add_to_* writes the bucket of # every non-blank field value. progress_block&.call(phase: :clearing, current: 0, total: field_values.size) live_bucket_keys = field_values.map { |value| send("#{index_name}_for", value).dbkey } MultiIndexGenerators.clear_index_buckets( dbclient, self, index_name, live_bucket_keys: live_bucket_keys, batch_size: batch_size ) do |key, cleared| progress_block&.call(phase: :clearing, current: cleared, total: field_values.size, key: key) end # PHASE 4: Rebuild index from cached objects (no reload needed) progress_block&.call(phase: :rebuilding, current: 0, total: cached_objects.size) processed = 0 cached_objects.each_slice(batch_size) do |objects| transaction do |_tx| objects.each do |obj| # Use the generated add_to method to maintain consistency # This ensures the same logic is used as during normal operation obj.send(:"add_to_#{scope_class_config}_#{index_name}", self) end end processed += objects.size progress_block&.call(phase: :rebuilding, current: processed, total: cached_objects.size) end Familia.info "[Rebuild] Multi-index #{index_name} rebuilt: #{field_values.size} field values, #{processed} objects" processed # Return count of processed objects else # No participation relationship found - warn and suggest alternative Familia.warn <<~WARNING [Rebuild] Cannot rebuild multi-index #{index_name}: no participation relationship found Multi-index rebuild requires a participation relationship to find objects. Add a participation relationship to #{indexed_class.name}: class #{indexed_class.name} < Familia::Horreum participates_in #{self.class.name}, :collection_name, score: :field end Then access the collection via: #{self.class.config_name}.collection_name WARNING nil end end end end |
.record_live_bucket_conflict(conflicts, owners, key, type, record_key) ⇒ void
This method returns an undefined value.
Adds +key+ to +conflicts+ for .check_live_bucket_keys! when it has more candidate owners than RecordKeyOwnership checks, is another record's key, or holds a type other than a set. For another record's key, +owners+ gets that record's hash key.
341 342 343 344 345 346 347 348 349 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 341 def record_live_bucket_conflict(conflicts, owners, key, type, record_key) if record_key == RecordKeyOwnership::TOO_MANY_CANDIDATES conflicts[key] = record_key return end owners[key] = record_key if record_key conflicts[key] = type if record_key || !WRITABLE_BUCKET_KEY_TYPES.include?(type) end |
.select_index_buckets(keys, bucket_prefix) ⇒ Array<String>
Returns the +keys+ that start with the literal +bucket_prefix+.
The check compares bytes. The client returns a key that is not valid UTF-8 as a binary String, and String#start_with? raises Encoding::CompatibilityError for a binary key and a UTF-8 prefix that both hold non-ASCII bytes, as they do when a scope identifier holds invalid UTF-8.
362 363 364 365 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 362 def select_index_buckets(keys, bucket_prefix) prefix = bucket_prefix.b keys.select { |key| key.b.start_with?(prefix) } end |
.setup(indexed_class:, field:, index_name:, within:, query:) ⇒ Object
Main setup method that orchestrates multi-value index creation
374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 374 def setup(indexed_class:, field:, index_name:, within:, query:) # Determine scope type: class-level or instance-scoped scope_class, scope_type = if within == :class [indexed_class, :class] else k = Familia.resolve_class(within) [k, :instance] end # Store metadata for this indexing relationship indexed_class.indexing_relationships << IndexingRelationship.new( field: field, scope_class: scope_class, within: within, # Preserve original (:class or actual class) index_name: index_name, query: query, cardinality: :multi, ) case scope_type when :instance # Instance-scoped multi-index (existing behavior) generate_factory_method(indexed_class, scope_class, index_name) generate_query_methods_destination(indexed_class, field, scope_class, index_name) if query generate_mutation_methods_self(indexed_class, field, scope_class, index_name) when :class # Class-level multi-index (new behavior) generate_factory_method_class(indexed_class, index_name) generate_query_methods_class(indexed_class, field, index_name) if query generate_mutation_methods_class(indexed_class, field, index_name) end end |
.validate_field_value(field_value, context: 'index') ⇒ String?
Validates a field value for use in index key construction. This is for data quality and debugging clarity. It does not make values safe to put in a key pattern: code that builds a SCAN or KEYS pattern from keys like these escapes the literal part with Familia.escape_glob (see .each_index_bucket_slice).
53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
# File 'lib/familia/features/relationships/indexing/multi_index_generators.rb', line 53 def validate_field_value(field_value, context: 'index') return nil if field_value.nil? str_value = field_value.to_s return nil if Familia.blank_text?(str_value) # The patterns below are ASCII, so a value whose bytes are not # valid in its encoding, which a regexp match raises on, is # matched as bytes. matchable = str_value.valid_encoding? ? str_value : str_value.b # Warn on values containing Redis glob pattern characters # These are legal but can be confusing when debugging key patterns if matchable.match?(/[*?\[\]]/) Familia.warn "[#{context}] Field value contains glob pattern characters: #{str_value.inspect}. " \ 'These are stored as literal characters but may be confusing during debugging.' end # Warn on control characters (except common whitespace) if matchable.match?(/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/) Familia.warn "[#{context}] Field value contains control characters: #{str_value.inspect}" end # Warn on excessively long values. The start of the value passes # through Familia.loggable_text: a BINARY slice with non-ASCII # bytes next to a non-ASCII class name in +context+ would raise # Encoding::CompatibilityError. if str_value.length > MAX_FIELD_VALUE_LENGTH Familia.warn "[#{context}] Field value exceeds #{MAX_FIELD_VALUE_LENGTH} characters " \ "(#{str_value.length} chars): #{Familia.loggable_text(str_value[0..50])}..." end str_value end |