Summary

InstCombine partially recognizes the common integer NaN test

(bitcast float to i32 & 0x7FFFFFFF) > 0x7F800000

but stops too early. It folds the sign-mask step to fabs, yet does not finish the job by converting the whole predicate to:

fcmp uno float %f, %f

That leaves x86 codegen at 4 instructions instead of the 2-instruction ucomiss plus setp sequence.

Reproducer

define i1 @float_is_nan(float %f) {
  %i = bitcast float %f to i32
  %a = and  i32 %i, 2147483647
  %r = icmp ugt i32 %a, 2139095040
  ret i1 %r
}

Optimized IR today

opt -O2 gets only part way there:

%1 = tail call float @llvm.fabs.f32(float %f)
%a = bitcast float %1 to i32
%r = icmp samesign ugt i32 %a, 2139095040
ret i1 %r

Current x86_64 output

movd   %xmm0, %eax
andl   $2147483647, %eax
cmpl   $2139095041, %eax
setae  %al
retq

Better output

ucomiss  %xmm0, %xmm0
setp     %al
retq

That is exactly what LLVM produces for:

%r = fcmp uno float %f, %f

Why this is valid

For IEEE single-precision:

  • 0x7F800000 is +inf
  • after stripping the sign bit, the only bit patterns larger than +inf are NaNs
  • fcmp uno %f, %f is true iff %f is NaN

So the two predicates are equivalent.

Why this seems worth tracking

  • InstCombine already sees part of the pattern
  • the final fold is crisp and target-independent
  • x86 has a very clean lowering for the floating-point form

Likely fix area

  • llvm/lib/Transforms/InstCombine/InstCombineAndOrXor.cpp
  • possibly InstCombineCasts.cpp

Local references

  • Source note: llvm-validation/ch17-floating-point/bug-float-nan-int-check.md
  • Harness: llvm-validation/ch17-floating-point/ch17_float.ll